2024-04-03 08:53:15,667 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-04-03 08:53:15,993 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-04-03 08:53:16,118 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-04-03 08:53:16,181 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 08:53:16,189 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-03 08:53:16,205 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=169, AvailableMemoryMB=11296
2024-04-03 08:53:16,207 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 08:53:16,207 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-03 08:53:16,354 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017592961:84 , srv1025328684:84 , srv1036725384:84 , srv1039462516:84 , srv1095172322:84 , srv109837694:84 , srv1118381060:84 , srv1132764522:84 , srv1142197110:84 , srv1163408198:84 , srv1176646607:84 , srv1181495846:84 , srv1185516964:84 , srv1186387462:84 , srv1245149254:84 , srv1262128344:84 , srv1268349463:84 , srv1314450849:84 , srv1318752329:84 , srv1327178761:84 , srv1376542905:84 , srv1396350747:84 , srv1404753740:84 , srv1431117263:84 , srv1446387857:84 , srv1483571821:84 , srv1484315881:84 , srv1488452108:84 , srv1504670351:84 , srv1535407429:84 , srv1535709009:84 , srv1545566749:84 , srv1550788134:84 , srv1562683619:84 , srv1565719060:84 , srv1568431990:84 , srv160699882:84 , srv162847368:84 , srv1637072547:84 , srv1646923585:84 , srv1679410480:84 , srv1690860628:84 , srv1699016568:84 , srv176631259:84 , srv1800662079:84 , srv1800865189:84 , srv1801193561:84 , srv1828041206:84 , srv1846582559:84 , srv1939645368:84 , srv194506438:84 , srv1966921285:84 , srv1983075023:84 , srv1984604263:84 , srv2002290361:84 , srv2036121689:84 , srv2036707739:84 , srv2065177755:84 , srv2087684686:84 , srv2106147065:84 , srv2134731982:84 , srv247238741:84 , srv252284835:84 , srv253088122:84 , srv285018630:84 , srv296780417:84 , srv330924783:84 , srv377726829:84 , srv398447321:84 , srv400571614:84 , srv401772250:84 , srv425558263:84 , srv439593210:84 , srv446538848:84 , srv474255120:84 , srv491141475:84 , srv497020354:84 , srv502680616:84 , srv573302817:84 , srv573725181:84 , srv602382432:84 , srv611047272:84 , srv611299241:84 , srv618672635:84 , srv657504097:84 , srv660500253:84 , srv666782409:84 , srv72546821:84 , srv732377595:84 , srv752555035:84 , srv760575150:84 , srv81543426:84 , srv831847476:84 , srv83791737:84 , srv848771480:84 , srv910582565:84 , srv926066595:84 , srv940950719:84 , srv945146305:84 , srv925321825:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-04-03 08:53:16,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:16,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318752329=18, srv81543426=91, srv1939645368=49, srv1535407429=29, srv666782409=86, srv1118381060=6, srv1095172322=4, srv194506438=50, srv1132764522=7, srv497020354=76, srv2134731982=60, srv502680616=77, srv1396350747=21, srv1404753740=22, srv1181495846=11, srv1690860628=41, srv491141475=75, srv732377595=88, srv474255120=74, srv1268349463=16, srv1185516964=12, srv1699016568=42, srv618672635=83, srv1984604263=53, srv253088122=63, srv446538848=73, srv1535709009=30, srv377726829=67, srv1966921285=51, srv660500253=85, srv330924783=66, srv439593210=72, srv1488452108=27, srv160699882=36, srv1039462516=3, srv1483571821=25, srv760575150=90, srv1446387857=24, srv1550788134=32, srv1017592961=0, srv1314450849=17, srv1983075023=52, srv573302817=78, srv848771480=94, srv2087684686=58, srv2036121689=55, srv1637072547=38, srv573725181=79, srv425558263=71, srv1025328684=1, srv1186387462=13, srv72546821=87, srv1036725384=2, srv247238741=61, srv925321825=96, srv1828041206=47, srv109837694=5, srv1800865189=45, srv296780417=65, srv1646923585=39, srv831847476=92, srv1431117263=23, srv400571614=69, srv2036707739=56, srv1504670351=28, srv1565719060=34, srv83791737=93, srv401772250=70, srv1800662079=44, srv1327178761=19, srv945146305=99, srv1484315881=26, srv1679410480=40, srv940950719=98, srv1562683619=33, srv611047272=81, srv1568431990=35, srv1545566749=31, srv398447321=68, srv926066595=97, srv2106147065=59, srv657504097=84, srv1163408198=9, srv1376542905=20, srv285018630=64, srv1262128344=15, srv252284835=62, srv162847368=37, srv752555035=89, srv2065177755=57, srv1176646607=10, srv1142197110=8, srv176631259=43, srv1801193561=46, srv602382432=80, srv611299241=82, srv2002290361=54, srv910582565=95, srv1245149254=14, srv1846582559=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 08:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 08:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 08:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 08:53:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 08:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 08:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 08:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 08:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 08:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 08:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 08:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 08:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 08:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 08:53:16,806 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:16,806 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 08:53:16,806 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 08:53:17,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1306 ms to try 5000 different iterations.  Found a solution that moves 4679 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18076837612005692. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03915525261325588, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924326383450113, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5198888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03915525261325588, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:17,868 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1446387857:81 , srv1431117263:82 , srv1637072547:82 , srv1939645368:82 , srv439593210:82 , srv945146305:82 , srv1132764522:83 , srv1828041206:83 , srv1984604263:83 , srv401772250:83 , srv474255120:83 , srv497020354:83 , srv1025328684:84 , srv1036725384:84 , srv1039462516:84 , srv1545566749:84 , srv162847368:84 , srv2036121689:84 , srv400571614:84 , srv611047272:84 , srv657504097:84 , srv666782409:84 , srv752555035:84 , srv1262128344:85 , srv1488452108:85 , srv1535407429:85 , srv1535709009:85 , srv1562683619:85 , srv176631259:85 , srv1846582559:85 , srv2106147065:85 , srv330924783:85 , srv377726829:85 , srv425558263:85 , srv502680616:85 , srv573302817:85 , srv618672635:85 , srv660500253:85 , srv72546821:85 , srv1376542905:86 , srv1404753740:86 , srv1565719060:86 , srv1568431990:86 , srv252284835:86 , srv602382432:86 , srv611299241:86 , srv910582565:86 , srv940950719:86 , srv1185516964:87 , srv1314450849:87 , srv1396350747:87 , srv1484315881:87 , srv1646923585:87 , srv1690860628:87 , srv1800662079:87 , srv1801193561:87 , srv1966921285:87 , srv2065177755:87 , srv2134731982:87 , srv247238741:87 , srv285018630:87 , srv732377595:87 , srv1095172322:88 , srv109837694:88 , srv1176646607:88 , srv1181495846:88 , srv1327178761:88 , srv1550788134:88 , srv1679410480:88 , srv2036707739:88 , srv253088122:88 , srv491141475:88 , srv573725181:88 , srv831847476:88 , srv1017592961:89 , srv1186387462:89 , srv1245149254:89 , srv1268349463:89 , srv1318752329:89 , srv1483571821:89 , srv1504670351:89 , srv160699882:89 , srv1699016568:89 , srv1983075023:89 , srv2002290361:89 , srv2087684686:89 , srv398447321:89 , srv446538848:89 , srv81543426:89 , srv83791737:89 , srv848771480:89 , srv1800865189:90 , srv194506438:90 , srv1118381060:91 , srv1142197110:91 , srv296780417:91 , srv926066595:91 , srv1163408198:92 , srv760575150:92 , srv925321825:438 }
2024-04-03 08:53:17,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:17,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318752329=18, srv81543426=91, srv1939645368=49, srv1535407429=29, srv666782409=86, srv1118381060=6, srv1095172322=4, srv194506438=50, srv1132764522=7, srv497020354=76, srv2134731982=60, srv502680616=77, srv1396350747=21, srv1404753740=22, srv1181495846=11, srv1690860628=41, srv491141475=75, srv732377595=88, srv474255120=74, srv1268349463=16, srv1185516964=12, srv1699016568=42, srv618672635=83, srv1984604263=53, srv253088122=63, srv446538848=73, srv1535709009=30, srv377726829=67, srv1966921285=51, srv660500253=85, srv330924783=66, srv439593210=72, srv1488452108=27, srv160699882=36, srv1039462516=3, srv1483571821=25, srv760575150=90, srv1446387857=24, srv1550788134=32, srv1017592961=0, srv1314450849=17, srv1983075023=52, srv573302817=78, srv848771480=94, srv2087684686=58, srv2036121689=55, srv1637072547=38, srv573725181=79, srv425558263=71, srv1025328684=1, srv1186387462=13, srv72546821=87, srv1036725384=2, srv247238741=61, srv925321825=96, srv1828041206=47, srv109837694=5, srv1800865189=45, srv296780417=65, srv1646923585=39, srv831847476=92, srv1431117263=23, srv400571614=69, srv2036707739=56, srv1504670351=28, srv1565719060=34, srv83791737=93, srv401772250=70, srv1800662079=44, srv1327178761=19, srv945146305=99, srv1484315881=26, srv1679410480=40, srv940950719=98, srv1562683619=33, srv611047272=81, srv1568431990=35, srv1545566749=31, srv398447321=68, srv926066595=97, srv2106147065=59, srv657504097=84, srv1163408198=9, srv1376542905=20, srv285018630=64, srv1262128344=15, srv252284835=62, srv162847368=37, srv752555035=89, srv2065177755=57, srv1176646607=10, srv1142197110=8, srv176631259=43, srv1801193561=46, srv602382432=80, srv611299241=82, srv2002290361=54, srv910582565=95, srv1245149254=14, srv1846582559=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:17,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:17,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:17,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:17,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:17,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:17,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:17,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:17,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 08:53:17,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 08:53:17,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 08:53:17,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 08:53:17,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 08:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 08:53:17,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 08:53:17,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 08:53:17,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 08:53:17,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 08:53:17,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 08:53:17,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 08:53:17,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 08:53:18,118 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03915525261325588, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924326383450113, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03915525261325588, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:18,119 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 08:53:18,119 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18073561306179806, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03915525261325588, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924326383450113, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03915525261325588, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 08:53:18,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 544 ms to try 5000 different iterations.  Found a solution that moves 2602 regions; Going from a computed imbalance of 0.18073561306179806 to a new imbalance of 0.01586712898292772. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021719912128406366); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599164387370716, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2891111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021719912128406366); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:18,500 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv760575150:78 , srv1118381060:79 , srv1163408198:80 , srv296780417:80 , srv1142197110:81 , srv2002290361:81 , srv2087684686:81 , srv109837694:82 , srv1327178761:82 , srv1483571821:82 , srv398447321:82 , srv1268349463:83 , srv1318752329:83 , srv1550788134:83 , srv160699882:83 , srv1800865189:83 , srv1983075023:83 , srv2036707739:83 , srv253088122:83 , srv83791737:83 , srv848771480:83 , srv926066595:83 , srv1017592961:84 , srv1095172322:84 , srv1176646607:84 , srv1181495846:84 , srv1314450849:84 , srv1376542905:84 , srv1484315881:84 , srv1504670351:84 , srv1646923585:84 , srv2065177755:84 , srv2134731982:84 , srv247238741:84 , srv611299241:84 , srv732377595:84 , srv81543426:84 , srv940950719:84 , srv1185516964:85 , srv1186387462:85 , srv1245149254:85 , srv1396350747:85 , srv1679410480:85 , srv1699016568:85 , srv1846582559:85 , srv194506438:85 , srv1966921285:85 , srv2036121689:85 , srv252284835:85 , srv910582565:85 , srv1404753740:86 , srv1488452108:86 , srv1535709009:86 , srv1801193561:86 , srv2106147065:86 , srv330924783:86 , srv446538848:86 , srv491141475:86 , srv502680616:86 , srv831847476:86 , srv1025328684:87 , srv1132764522:87 , srv1562683619:87 , srv1565719060:87 , srv1568431990:87 , srv162847368:87 , srv1690860628:87 , srv285018630:87 , srv377726829:87 , srv425558263:87 , srv573302817:87 , srv573725181:87 , srv602382432:87 , srv611047272:87 , srv752555035:87 , srv1039462516:88 , srv1535407429:88 , srv474255120:88 , srv618672635:88 , srv660500253:88 , srv666782409:88 , srv72546821:88 , srv1036725384:89 , srv1262128344:89 , srv1637072547:89 , srv1800662079:89 , srv1828041206:89 , srv1984604263:89 , srv400571614:89 , srv439593210:89 , srv945146305:89 , srv1545566749:90 , srv1939645368:90 , srv401772250:90 , srv497020354:90 , srv657504097:90 , srv1446387857:91 , srv176631259:91 , srv1431117263:92 , srv925321825:529 }
2024-04-03 08:53:18,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:18,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318752329=18, srv81543426=91, srv1939645368=49, srv1535407429=29, srv666782409=86, srv1118381060=6, srv1095172322=4, srv194506438=50, srv1132764522=7, srv497020354=76, srv2134731982=60, srv502680616=77, srv1396350747=21, srv1404753740=22, srv1181495846=11, srv1690860628=41, srv491141475=75, srv732377595=88, srv474255120=74, srv1268349463=16, srv1185516964=12, srv1699016568=42, srv618672635=83, srv1984604263=53, srv253088122=63, srv446538848=73, srv1535709009=30, srv377726829=67, srv1966921285=51, srv660500253=85, srv330924783=66, srv439593210=72, srv1488452108=27, srv160699882=36, srv1039462516=3, srv1483571821=25, srv760575150=90, srv1446387857=24, srv1550788134=32, srv1017592961=0, srv1314450849=17, srv1983075023=52, srv573302817=78, srv848771480=94, srv2087684686=58, srv2036121689=55, srv1637072547=38, srv573725181=79, srv425558263=71, srv1025328684=1, srv1186387462=13, srv72546821=87, srv1036725384=2, srv247238741=61, srv925321825=96, srv1828041206=47, srv109837694=5, srv1800865189=45, srv296780417=65, srv1646923585=39, srv831847476=92, srv1431117263=23, srv400571614=69, srv2036707739=56, srv1504670351=28, srv1565719060=34, srv83791737=93, srv401772250=70, srv1800662079=44, srv1327178761=19, srv945146305=99, srv1484315881=26, srv1679410480=40, srv940950719=98, srv1562683619=33, srv611047272=81, srv1568431990=35, srv1545566749=31, srv398447321=68, srv926066595=97, srv2106147065=59, srv657504097=84, srv1163408198=9, srv1376542905=20, srv285018630=64, srv1262128344=15, srv252284835=62, srv162847368=37, srv752555035=89, srv2065177755=57, srv1176646607=10, srv1142197110=8, srv176631259=43, srv1801193561=46, srv602382432=80, srv611299241=82, srv2002290361=54, srv910582565=95, srv1245149254=14, srv1846582559=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:18,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 08:53:18,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 08:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 08:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 08:53:18,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 08:53:18,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 08:53:18,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 08:53:18,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 08:53:18,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 08:53:18,671 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021719912128406366); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599164387370716, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021719912128406366); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:18,671 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 08:53:18,672 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01584890938951254, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021719912128406366); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03599164387370716, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021719912128406366); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 08:53:18,985 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations.  Found a solution that moves 978 regions; Going from a computed imbalance of 0.01584890938951254 to a new imbalance of 7.609261944543917E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005216356521416409); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008606629658238704); MoveCostFunction : (multiplier=7.0, imbalance=0.10866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005216356521416409); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:18,987 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv176631259:82 , srv1800662079:82 , srv194506438:82 , srv657504097:82 , srv1186387462:83 , srv1245149254:83 , srv1699016568:83 , srv285018630:83 , srv446538848:83 , srv573725181:83 , srv831847476:83 , srv1017592961:84 , srv1262128344:84 , srv1504670351:84 , srv1545566749:84 , srv1565719060:84 , srv1568431990:84 , srv1679410480:84 , srv491141475:84 , srv497020354:84 , srv618672635:84 , srv72546821:84 , srv81543426:84 , srv926066595:84 , srv1036725384:85 , srv1142197110:85 , srv1181495846:85 , srv1396350747:85 , srv1446387857:85 , srv1535407429:85 , srv1690860628:85 , srv1800865189:85 , srv1939645368:85 , srv1983075023:85 , srv1984604263:85 , srv377726829:85 , srv400571614:85 , srv602382432:85 , srv660500253:85 , srv666782409:85 , srv83791737:85 , srv1025328684:86 , srv1039462516:86 , srv1095172322:86 , srv1163408198:86 , srv1176646607:86 , srv1185516964:86 , srv1268349463:86 , srv1314450849:86 , srv1318752329:86 , srv1431117263:86 , srv1483571821:86 , srv1484315881:86 , srv1488452108:86 , srv1535709009:86 , srv1550788134:86 , srv1562683619:86 , srv162847368:86 , srv1637072547:86 , srv1801193561:86 , srv1828041206:86 , srv1966921285:86 , srv2036707739:86 , srv2106147065:86 , srv2134731982:86 , srv253088122:86 , srv330924783:86 , srv401772250:86 , srv425558263:86 , srv439593210:86 , srv502680616:86 , srv573302817:86 , srv732377595:86 , srv848771480:86 , srv945146305:86 , srv109837694:87 , srv1132764522:87 , srv1327178761:87 , srv1376542905:87 , srv1404753740:87 , srv160699882:87 , srv1646923585:87 , srv1846582559:87 , srv2002290361:87 , srv2087684686:87 , srv247238741:87 , srv252284835:87 , srv296780417:87 , srv398447321:87 , srv474255120:87 , srv611047272:87 , srv611299241:87 , srv752555035:87 , srv910582565:87 , srv1118381060:88 , srv2036121689:88 , srv2065177755:88 , srv760575150:88 , srv940950719:88 , srv925321825:537 }
2024-04-03 08:53:18,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:18,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318752329=18, srv81543426=91, srv1939645368=49, srv1535407429=29, srv666782409=86, srv1118381060=6, srv1095172322=4, srv194506438=50, srv1132764522=7, srv497020354=76, srv2134731982=60, srv502680616=77, srv1396350747=21, srv1404753740=22, srv1181495846=11, srv1690860628=41, srv491141475=75, srv732377595=88, srv474255120=74, srv1268349463=16, srv1185516964=12, srv1699016568=42, srv618672635=83, srv1984604263=53, srv253088122=63, srv446538848=73, srv1535709009=30, srv377726829=67, srv1966921285=51, srv660500253=85, srv330924783=66, srv439593210=72, srv1488452108=27, srv160699882=36, srv1039462516=3, srv1483571821=25, srv760575150=90, srv1446387857=24, srv1550788134=32, srv1017592961=0, srv1314450849=17, srv1983075023=52, srv573302817=78, srv848771480=94, srv2087684686=58, srv2036121689=55, srv1637072547=38, srv573725181=79, srv425558263=71, srv1025328684=1, srv1186387462=13, srv72546821=87, srv1036725384=2, srv247238741=61, srv925321825=96, srv1828041206=47, srv109837694=5, srv1800865189=45, srv296780417=65, srv1646923585=39, srv831847476=92, srv1431117263=23, srv400571614=69, srv2036707739=56, srv1504670351=28, srv1565719060=34, srv83791737=93, srv401772250=70, srv1800662079=44, srv1327178761=19, srv945146305=99, srv1484315881=26, srv1679410480=40, srv940950719=98, srv1562683619=33, srv611047272=81, srv1568431990=35, srv1545566749=31, srv398447321=68, srv926066595=97, srv2106147065=59, srv657504097=84, srv1163408198=9, srv1376542905=20, srv285018630=64, srv1262128344=15, srv252284835=62, srv162847368=37, srv752555035=89, srv2065177755=57, srv1176646607=10, srv1142197110=8, srv176631259=43, srv1801193561=46, srv602382432=80, srv611299241=82, srv2002290361=54, srv910582565=95, srv1245149254=14, srv1846582559=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:19,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 08:53:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 08:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 08:53:19,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 08:53:19,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 08:53:19,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 08:53:19,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 08:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 08:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 08:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 08:53:19,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 08:53:19,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005216356521416409); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008606629658238704); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005216356521416409); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:19,156 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 08:53:19,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.54078092086965E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005216356521416409); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008606629658238704); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005216356521416409); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 08:53:19,473 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations.  Found a solution that moves 250 regions; Going from a computed imbalance of 7.54078092086965E-4 to a new imbalance of 1.1303162137360209E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.027777777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:19,474 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1431117263:82 , srv1404753740:83 , srv160699882:83 , srv1690860628:83 , srv2065177755:83 , srv401772250:83 , srv1036725384:84 , srv1095172322:84 , srv1118381060:84 , srv1176646607:84 , srv1185516964:84 , srv1262128344:84 , srv1268349463:84 , srv1318752329:84 , srv1376542905:84 , srv1535407429:84 , srv1545566749:84 , srv1562683619:84 , srv1637072547:84 , srv1646923585:84 , srv1679410480:84 , srv176631259:84 , srv1800662079:84 , srv1800865189:84 , srv1801193561:84 , srv1828041206:84 , srv1846582559:84 , srv1966921285:84 , srv247238741:84 , srv252284835:84 , srv296780417:84 , srv398447321:84 , srv400571614:84 , srv425558263:84 , srv439593210:84 , srv446538848:84 , srv474255120:84 , srv491141475:84 , srv573302817:84 , srv573725181:84 , srv602382432:84 , srv660500253:84 , srv752555035:84 , srv831847476:84 , srv848771480:84 , srv910582565:84 , srv926066595:84 , srv940950719:84 , srv1017592961:85 , srv1025328684:85 , srv1039462516:85 , srv109837694:85 , srv1132764522:85 , srv1142197110:85 , srv1163408198:85 , srv1181495846:85 , srv1186387462:85 , srv1245149254:85 , srv1314450849:85 , srv1327178761:85 , srv1396350747:85 , srv1446387857:85 , srv1483571821:85 , srv1484315881:85 , srv1488452108:85 , srv1504670351:85 , srv1535709009:85 , srv1550788134:85 , srv1565719060:85 , srv1568431990:85 , srv162847368:85 , srv1699016568:85 , srv1939645368:85 , srv194506438:85 , srv1983075023:85 , srv1984604263:85 , srv2002290361:85 , srv2036121689:85 , srv2036707739:85 , srv2087684686:85 , srv2106147065:85 , srv2134731982:85 , srv253088122:85 , srv285018630:85 , srv330924783:85 , srv377726829:85 , srv497020354:85 , srv502680616:85 , srv611047272:85 , srv611299241:85 , srv618672635:85 , srv666782409:85 , srv72546821:85 , srv732377595:85 , srv760575150:85 , srv81543426:85 , srv83791737:85 , srv657504097:86 , srv945146305:86 , srv925321825:638 }
2024-04-03 08:53:19,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:19,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318752329=18, srv81543426=91, srv1939645368=49, srv1535407429=29, srv666782409=86, srv1118381060=6, srv1095172322=4, srv194506438=50, srv1132764522=7, srv497020354=76, srv2134731982=60, srv502680616=77, srv1396350747=21, srv1404753740=22, srv1181495846=11, srv1690860628=41, srv491141475=75, srv732377595=88, srv474255120=74, srv1268349463=16, srv1185516964=12, srv1699016568=42, srv618672635=83, srv1984604263=53, srv253088122=63, srv446538848=73, srv1535709009=30, srv377726829=67, srv1966921285=51, srv660500253=85, srv330924783=66, srv439593210=72, srv1488452108=27, srv160699882=36, srv1039462516=3, srv1483571821=25, srv760575150=90, srv1446387857=24, srv1550788134=32, srv1017592961=0, srv1314450849=17, srv1983075023=52, srv573302817=78, srv848771480=94, srv2087684686=58, srv2036121689=55, srv1637072547=38, srv573725181=79, srv425558263=71, srv1025328684=1, srv1186387462=13, srv72546821=87, srv1036725384=2, srv247238741=61, srv925321825=96, srv1828041206=47, srv109837694=5, srv1800865189=45, srv296780417=65, srv1646923585=39, srv831847476=92, srv1431117263=23, srv400571614=69, srv2036707739=56, srv1504670351=28, srv1565719060=34, srv83791737=93, srv401772250=70, srv1800662079=44, srv1327178761=19, srv945146305=99, srv1484315881=26, srv1679410480=40, srv940950719=98, srv1562683619=33, srv611047272=81, srv1568431990=35, srv1545566749=31, srv398447321=68, srv926066595=97, srv2106147065=59, srv657504097=84, srv1163408198=9, srv1376542905=20, srv285018630=64, srv1262128344=15, srv252284835=62, srv162847368=37, srv752555035=89, srv2065177755=57, srv1176646607=10, srv1142197110=8, srv176631259=43, srv1801193561=46, srv602382432=80, srv611299241=82, srv2002290361=54, srv910582565=95, srv1245149254=14, srv1846582559=48} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:19,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 08:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 08:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 08:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 08:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 08:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 08:53:19,605 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.552624722373809E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:19,606 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1431117263:82 , srv1404753740:83 , srv160699882:83 , srv1690860628:83 , srv2065177755:83 , srv401772250:83 , srv1036725384:84 , srv1095172322:84 , srv1118381060:84 , srv1176646607:84 , srv1185516964:84 , srv1262128344:84 , srv1268349463:84 , srv1318752329:84 , srv1376542905:84 , srv1535407429:84 , srv1545566749:84 , srv1562683619:84 , srv1637072547:84 , srv1646923585:84 , srv1679410480:84 , srv176631259:84 , srv1800662079:84 , srv1800865189:84 , srv1801193561:84 , srv1828041206:84 , srv1846582559:84 , srv1966921285:84 , srv247238741:84 , srv252284835:84 , srv296780417:84 , srv398447321:84 , srv400571614:84 , srv425558263:84 , srv439593210:84 , srv446538848:84 , srv474255120:84 , srv491141475:84 , srv573302817:84 , srv573725181:84 , srv602382432:84 , srv660500253:84 , srv752555035:84 , srv831847476:84 , srv848771480:84 , srv910582565:84 , srv926066595:84 , srv940950719:84 , srv1017592961:85 , srv1025328684:85 , srv1039462516:85 , srv109837694:85 , srv1132764522:85 , srv1142197110:85 , srv1163408198:85 , srv1181495846:85 , srv1186387462:85 , srv1245149254:85 , srv1314450849:85 , srv1327178761:85 , srv1396350747:85 , srv1446387857:85 , srv1483571821:85 , srv1484315881:85 , srv1488452108:85 , srv1504670351:85 , srv1535709009:85 , srv1550788134:85 , srv1565719060:85 , srv1568431990:85 , srv162847368:85 , srv1699016568:85 , srv1939645368:85 , srv194506438:85 , srv1983075023:85 , srv1984604263:85 , srv2002290361:85 , srv2036121689:85 , srv2036707739:85 , srv2087684686:85 , srv2106147065:85 , srv2134731982:85 , srv253088122:85 , srv285018630:85 , srv330924783:85 , srv377726829:85 , srv497020354:85 , srv502680616:85 , srv611047272:85 , srv611299241:85 , srv618672635:85 , srv666782409:85 , srv72546821:85 , srv732377595:85 , srv760575150:85 , srv81543426:85 , srv83791737:85 , srv657504097:86 , srv945146305:86 , srv925321825:638 }
2024-04-03 08:53:19,634 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=275 (was 273) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=10530 (was 11296)
2024-04-03 08:53:19,645 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=169, AvailableMemoryMB=10530
2024-04-03 08:53:19,646 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 08:53:19,646 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-04-03 08:53:19,654 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001438597:84 , srv1005456907:84 , srv1027172407:84 , srv1064411436:84 , srv1127544335:84 , srv1130887585:84 , srv1155892557:84 , srv1211247267:84 , srv122128516:84 , srv130870903:84 , srv1401590150:84 , srv1455797338:84 , srv1503772351:84 , srv1513112391:84 , srv1753074977:84 , srv182138290:84 , srv1957696278:84 , srv2002225406:84 , srv2004491002:84 , srv2040792523:84 , srv2107063303:84 , srv491988666:84 , srv517001384:84 , srv529923781:84 , srv670540558:84 , srv748179928:84 , srv827560522:84 , srv861934799:84 , srv899669200:84 , srv1634113362:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-04-03 08:53:19,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:53:19,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1130887585=5, srv1155892557=6, srv1001438597=0, srv1957696278=17, srv1513112391=13, srv899669200=29, srv1005456907=1, srv130870903=9, srv1753074977=15, srv1401590150=10, srv1211247267=7, srv1064411436=3, srv2040792523=20, srv2107063303=21, srv1634113362=14, srv2004491002=19, srv122128516=8, srv182138290=16, srv491988666=22, srv529923781=24, srv670540558=25, srv827560522=27, srv1455797338=11, srv861934799=28, srv1503772351=12, srv2002225406=18, srv1027172407=2, srv517001384=23, srv748179928=26, srv1127544335=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:53:19,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:53:19,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:53:19,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:53:19,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:53:19,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:53:19,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:53:19,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:53:19,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:53:19,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:53:19,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-03 08:53:19,679 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.142950068023009, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:53:19,679 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991264347426087, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.142950068023009, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-04-03 08:55:19,668 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6261576 different iterations.  Found a solution that moves 1858 regions; Going from a computed imbalance of 0.991264347426087 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:55:19,674 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1001438597:90 , srv1005456907:90 , srv1027172407:90 , srv1064411436:90 , srv1127544335:90 , srv1130887585:90 , srv1155892557:90 , srv1211247267:90 , srv122128516:90 , srv130870903:90 , srv1401590150:90 , srv1455797338:90 , srv1503772351:90 , srv1513112391:90 , srv1634113362:90 , srv1753074977:90 , srv182138290:90 , srv1957696278:90 , srv2002225406:90 , srv2004491002:90 , srv2040792523:90 , srv2107063303:90 , srv491988666:90 , srv517001384:90 , srv529923781:90 , srv670540558:90 , srv748179928:90 , srv827560522:90 , srv861934799:90 , srv899669200:90 }
2024-04-03 08:55:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 08:55:19,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1130887585=5, srv1155892557=6, srv1001438597=0, srv1957696278=17, srv1513112391=13, srv899669200=29, srv1005456907=1, srv130870903=9, srv1753074977=15, srv1401590150=10, srv1211247267=7, srv1064411436=3, srv2040792523=20, srv2107063303=21, srv1634113362=14, srv2004491002=19, srv122128516=8, srv182138290=16, srv491988666=22, srv529923781=24, srv670540558=25, srv827560522=27, srv1455797338=11, srv861934799=28, srv1503772351=12, srv2002225406=18, srv1027172407=2, srv517001384=23, srv748179928=26, srv1127544335=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 08:55:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 08:55:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 08:55:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 08:55:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 08:55:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 08:55:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 08:55:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 08:55:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 08:55:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-03 08:55:19,692 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 08:55:19,692 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1001438597:90 , srv1005456907:90 , srv1027172407:90 , srv1064411436:90 , srv1127544335:90 , srv1130887585:90 , srv1155892557:90 , srv1211247267:90 , srv122128516:90 , srv130870903:90 , srv1401590150:90 , srv1455797338:90 , srv1503772351:90 , srv1513112391:90 , srv1634113362:90 , srv1753074977:90 , srv182138290:90 , srv1957696278:90 , srv2002225406:90 , srv2004491002:90 , srv2040792523:90 , srv2107063303:90 , srv491988666:90 , srv517001384:90 , srv529923781:90 , srv670540558:90 , srv748179928:90 , srv827560522:90 , srv861934799:90 , srv899669200:90 }
2024-04-03 08:55:19,706 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=182 (was 275), ProcessCount=167 (was 169), AvailableMemoryMB=9658 (was 10530)