2024-01-03 07:53:15,465 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-03 07:53:15,790 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-03 07:53:15,920 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-03 07:53:15,994 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 07:53:16,001 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-03 07:53:16,015 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=274, ProcessCount=167, AvailableMemoryMB=3762
2024-01-03 07:53:16,017 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 07:53:16,018 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-03 07:53:16,164 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101051400:84 , srv1022143361:84 , srv1039944822:84 , srv1045393640:84 , srv1078921953:84 , srv1083896039:84 , srv1091230119:84 , srv1093488197:84 , srv1123308605:84 , srv1133656394:84 , srv115252029:84 , srv1161100913:84 , srv1170923191:84 , srv1201490298:84 , srv1227236776:84 , srv1267386458:84 , srv1292145949:84 , srv1298388028:84 , srv1313082732:84 , srv133175769:84 , srv13474807:84 , srv136939393:84 , srv137667465:84 , srv1411834182:84 , srv1419785319:84 , srv1490330925:84 , srv1536927478:84 , srv1568068485:84 , srv1573704359:84 , srv1608298365:84 , srv1649249172:84 , srv1651458744:84 , srv1653424303:84 , srv16558298:84 , srv170133179:84 , srv1730205604:84 , srv1746303627:84 , srv1757453936:84 , srv1761498848:84 , srv1767582992:84 , srv1801515376:84 , srv1811824875:84 , srv1875190880:84 , srv1883862821:84 , srv1895006385:84 , srv1908426365:84 , srv1936529187:84 , srv1938366169:84 , srv1996352230:84 , srv2021207184:84 , srv2041027691:84 , srv2054578763:84 , srv2057505091:84 , srv2063301828:84 , srv2101252202:84 , srv2140307477:84 , srv221933925:84 , srv249030372:84 , srv252592563:84 , srv258178445:84 , srv281911471:84 , srv282935901:84 , srv297458423:84 , srv300409840:84 , srv303217466:84 , srv328021826:84 , srv347782304:84 , srv365084930:84 , srv367589924:84 , srv385959515:84 , srv402003910:84 , srv403866941:84 , srv435557300:84 , srv448501426:84 , srv498049780:84 , srv506095823:84 , srv515961396:84 , srv533290141:84 , srv615894080:84 , srv633786773:84 , srv63403952:84 , srv659715862:84 , srv666640949:84 , srv680801522:84 , srv717408535:84 , srv723660757:84 , srv752825282:84 , srv762262772:84 , srv778156912:84 , srv827237230:84 , srv856598904:84 , srv873238380:84 , srv879517303:84 , srv894257379:84 , srv897500745:84 , srv906146083:84 , srv907185366:84 , srv923504960:84 , srv97260617:84 , srv1501349498:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-03 07:53:16,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:16,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1292145949=16, srv873238380=92, srv1083896039=5, srv1039944822=2, srv723660757=86, srv1490330925=25, srv762262772=88, srv1123308605=8, srv1746303627=37, srv1501349498=26, srv282935901=62, srv367589924=69, srv435557300=73, srv778156912=89, srv1608298365=30, srv63403952=81, srv365084930=68, srv1811824875=42, srv1895006385=45, srv249030372=58, srv1161100913=11, srv1757453936=38, srv1201490298=13, srv1883862821=44, srv136939393=21, srv856598904=91, srv403866941=72, srv615894080=79, srv659715862=82, srv1653424303=33, srv2041027691=51, srv221933925=57, srv2063301828=54, srv498049780=75, srv907185366=97, srv1938366169=48, srv515961396=77, srv717408535=85, srv533290141=78, srv97260617=99, srv1045393640=3, srv2057505091=53, srv1313082732=18, srv1093488197=7, srv1649249172=31, srv752825282=87, srv923504960=98, srv1573704359=29, srv1227236776=14, srv133175769=19, srv300409840=64, srv633786773=80, srv16558298=34, srv101051400=0, srv281911471=61, srv1419785319=24, srv906146083=96, srv1767582992=40, srv347782304=67, srv1022143361=1, srv402003910=71, srv897500745=95, srv2021207184=50, srv1730205604=36, srv879517303=93, srv1091230119=6, srv303217466=65, srv1875190880=43, srv1536927478=27, srv115252029=10, srv328021826=66, srv258178445=60, srv137667465=22, srv170133179=35, srv297458423=63, srv252592563=59, srv506095823=76, srv1078921953=4, srv448501426=74, srv894257379=94, srv2140307477=56, srv1170923191=12, srv2101252202=55, srv1761498848=39, srv1996352230=49, srv1936529187=47, srv1411834182=23, srv680801522=84, srv1801515376=41, srv1298388028=17, srv385959515=70, srv1133656394=9, srv1267386458=15, srv1651458744=32, srv13474807=20, srv1568068485=28, srv2054578763=52, srv1908426365=46, srv666640949=83, srv827237230=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:16,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:16,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:16,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:16,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:16,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 07:53:16,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 07:53:16,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 07:53:16,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 07:53:16,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 07:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 07:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 07:53:16,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 07:53:16,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 07:53:16,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 07:53:16,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 07:53:16,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 07:53:16,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 07:53:16,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 07:53:16,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 07:53:16,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 07:53:16,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 07:53:16,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 07:53:16,728 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:16,728 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 07:53:16,729 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 07:53:17,135 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 810 ms to try 5000 different iterations.  Found a solution that moves 4724 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1766406827765971. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041714114398694116, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05611836136777174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5248888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041714114398694116, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:17,158 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1045393640:80 , srv402003910:81 , srv1083896039:82 , srv1490330925:82 , srv1767582992:82 , srv2041027691:82 , srv762262772:82 , srv1170923191:83 , srv1573704359:83 , srv1801515376:83 , srv752825282:83 , srv1936529187:84 , srv1938366169:84 , srv2063301828:84 , srv2101252202:84 , srv385959515:84 , srv856598904:84 , srv907185366:84 , srv97260617:84 , srv1078921953:85 , srv115252029:85 , srv1292145949:85 , srv13474807:85 , srv1761498848:85 , srv1811824875:85 , srv1996352230:85 , srv365084930:85 , srv367589924:85 , srv435557300:85 , srv498049780:85 , srv633786773:85 , srv63403952:85 , srv680801522:85 , srv723660757:85 , srv778156912:85 , srv101051400:86 , srv1022143361:86 , srv1039944822:86 , srv1161100913:86 , srv137667465:86 , srv1411834182:86 , srv1419785319:86 , srv1649249172:86 , srv1651458744:86 , srv16558298:86 , srv1730205604:86 , srv2054578763:86 , srv328021826:86 , srv506095823:86 , srv533290141:86 , srv659715862:86 , srv717408535:86 , srv827237230:86 , srv894257379:86 , srv1091230119:87 , srv1093488197:87 , srv1227236776:87 , srv133175769:87 , srv1608298365:87 , srv1746303627:87 , srv1757453936:87 , srv2140307477:87 , srv282935901:87 , srv297458423:87 , srv303217466:87 , srv448501426:87 , srv515961396:87 , srv615894080:87 , srv879517303:87 , srv923504960:87 , srv1123308605:88 , srv1133656394:88 , srv1313082732:88 , srv136939393:88 , srv1568068485:88 , srv1653424303:88 , srv1875190880:88 , srv1895006385:88 , srv221933925:88 , srv249030372:88 , srv252592563:88 , srv347782304:88 , srv906146083:88 , srv1536927478:89 , srv1883862821:89 , srv1908426365:89 , srv2021207184:89 , srv2057505091:89 , srv281911471:89 , srv300409840:89 , srv897500745:89 , srv1267386458:90 , srv170133179:90 , srv258178445:90 , srv403866941:90 , srv666640949:90 , srv873238380:90 , srv1201490298:91 , srv1298388028:91 , srv1501349498:461 }
2024-01-03 07:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1292145949=16, srv873238380=92, srv1083896039=5, srv1039944822=2, srv723660757=86, srv1490330925=25, srv762262772=88, srv1123308605=8, srv1746303627=37, srv1501349498=26, srv282935901=62, srv367589924=69, srv435557300=73, srv778156912=89, srv1608298365=30, srv63403952=81, srv365084930=68, srv1811824875=42, srv1895006385=45, srv249030372=58, srv1161100913=11, srv1757453936=38, srv1201490298=13, srv1883862821=44, srv136939393=21, srv856598904=91, srv403866941=72, srv615894080=79, srv659715862=82, srv1653424303=33, srv2041027691=51, srv221933925=57, srv2063301828=54, srv498049780=75, srv907185366=97, srv1938366169=48, srv515961396=77, srv717408535=85, srv533290141=78, srv97260617=99, srv1045393640=3, srv2057505091=53, srv1313082732=18, srv1093488197=7, srv1649249172=31, srv752825282=87, srv923504960=98, srv1573704359=29, srv1227236776=14, srv133175769=19, srv300409840=64, srv633786773=80, srv16558298=34, srv101051400=0, srv281911471=61, srv1419785319=24, srv906146083=96, srv1767582992=40, srv347782304=67, srv1022143361=1, srv402003910=71, srv897500745=95, srv2021207184=50, srv1730205604=36, srv879517303=93, srv1091230119=6, srv303217466=65, srv1875190880=43, srv1536927478=27, srv115252029=10, srv328021826=66, srv258178445=60, srv137667465=22, srv170133179=35, srv297458423=63, srv252592563=59, srv506095823=76, srv1078921953=4, srv448501426=74, srv894257379=94, srv2140307477=56, srv1170923191=12, srv2101252202=55, srv1761498848=39, srv1996352230=49, srv1936529187=47, srv1411834182=23, srv680801522=84, srv1801515376=41, srv1298388028=17, srv385959515=70, srv1133656394=9, srv1267386458=15, srv1651458744=32, srv13474807=20, srv1568068485=28, srv2054578763=52, srv1908426365=46, srv666640949=83, srv827237230=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:17,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:17,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 07:53:17,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 07:53:17,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 07:53:17,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 07:53:17,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 07:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 07:53:17,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 07:53:17,328 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041714114398694116, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05611836136777174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041714114398694116, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:17,328 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 07:53:17,328 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17660760462160355, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041714114398694116, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05611836136777174, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041714114398694116, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 07:53:17,565 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 346 ms to try 5000 different iterations.  Found a solution that moves 2643 regions; Going from a computed imbalance of 0.17660760462160355 to a new imbalance of 0.014062539181567657. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022046442833282588); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0313483021770353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2936666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022046442833282588); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:17,575 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1201490298:80 , srv1267386458:81 , srv1298388028:81 , srv1313082732:82 , srv1875190880:82 , srv403866941:82 , srv666640949:82 , srv906146083:82 , srv170133179:83 , srv1883862821:83 , srv1895006385:83 , srv1908426365:83 , srv2057505091:83 , srv300409840:83 , srv303217466:83 , srv347782304:83 , srv533290141:83 , srv897500745:83 , srv1091230119:84 , srv1133656394:84 , srv1227236776:84 , srv136939393:84 , srv1536927478:84 , srv1653424303:84 , srv1730205604:84 , srv1746303627:84 , srv2021207184:84 , srv249030372:84 , srv281911471:84 , srv448501426:84 , srv873238380:84 , srv101051400:85 , srv1093488197:85 , srv1123308605:85 , srv137667465:85 , srv1757453936:85 , srv2054578763:85 , srv2140307477:85 , srv252592563:85 , srv258178445:85 , srv498049780:85 , srv659715862:85 , srv923504960:85 , srv97260617:85 , srv1022143361:86 , srv1039944822:86 , srv1078921953:86 , srv115252029:86 , srv1161100913:86 , srv133175769:86 , srv13474807:86 , srv1608298365:86 , srv1649249172:86 , srv1651458744:86 , srv2101252202:86 , srv282935901:86 , srv297458423:86 , srv367589924:86 , srv506095823:86 , srv633786773:86 , srv723660757:86 , srv827237230:86 , srv879517303:86 , srv894257379:86 , srv1419785319:87 , srv16558298:87 , srv1936529187:87 , srv1996352230:87 , srv365084930:87 , srv435557300:87 , srv515961396:87 , srv615894080:87 , srv63403952:87 , srv680801522:87 , srv778156912:87 , srv1083896039:88 , srv1170923191:88 , srv1292145949:88 , srv1411834182:88 , srv1568068485:88 , srv1761498848:88 , srv1938366169:88 , srv221933925:88 , srv328021826:88 , srv717408535:88 , srv752825282:88 , srv907185366:88 , srv1573704359:89 , srv1801515376:89 , srv1811824875:89 , srv2063301828:89 , srv762262772:89 , srv856598904:89 , srv1490330925:90 , srv1767582992:90 , srv402003910:90 , srv385959515:91 , srv1045393640:92 , srv2041027691:92 , srv1501349498:509 }
2024-01-03 07:53:17,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:17,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1292145949=16, srv873238380=92, srv1083896039=5, srv1039944822=2, srv723660757=86, srv1490330925=25, srv762262772=88, srv1123308605=8, srv1746303627=37, srv1501349498=26, srv282935901=62, srv367589924=69, srv435557300=73, srv778156912=89, srv1608298365=30, srv63403952=81, srv365084930=68, srv1811824875=42, srv1895006385=45, srv249030372=58, srv1161100913=11, srv1757453936=38, srv1201490298=13, srv1883862821=44, srv136939393=21, srv856598904=91, srv403866941=72, srv615894080=79, srv659715862=82, srv1653424303=33, srv2041027691=51, srv221933925=57, srv2063301828=54, srv498049780=75, srv907185366=97, srv1938366169=48, srv515961396=77, srv717408535=85, srv533290141=78, srv97260617=99, srv1045393640=3, srv2057505091=53, srv1313082732=18, srv1093488197=7, srv1649249172=31, srv752825282=87, srv923504960=98, srv1573704359=29, srv1227236776=14, srv133175769=19, srv300409840=64, srv633786773=80, srv16558298=34, srv101051400=0, srv281911471=61, srv1419785319=24, srv906146083=96, srv1767582992=40, srv347782304=67, srv1022143361=1, srv402003910=71, srv897500745=95, srv2021207184=50, srv1730205604=36, srv879517303=93, srv1091230119=6, srv303217466=65, srv1875190880=43, srv1536927478=27, srv115252029=10, srv328021826=66, srv258178445=60, srv137667465=22, srv170133179=35, srv297458423=63, srv252592563=59, srv506095823=76, srv1078921953=4, srv448501426=74, srv894257379=94, srv2140307477=56, srv1170923191=12, srv2101252202=55, srv1761498848=39, srv1996352230=49, srv1936529187=47, srv1411834182=23, srv680801522=84, srv1801515376=41, srv1298388028=17, srv385959515=70, srv1133656394=9, srv1267386458=15, srv1651458744=32, srv13474807=20, srv1568068485=28, srv2054578763=52, srv1908426365=46, srv666640949=83, srv827237230=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:17,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:17,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:17,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 07:53:17,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 07:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 07:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 07:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 07:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 07:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 07:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 07:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 07:53:17,726 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022046442833282588); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0313483021770353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022046442833282588); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:17,727 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 07:53:17,727 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014044032500016422, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022046442833282588); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0313483021770353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022046442833282588); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 07:53:17,955 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 327 ms to try 5000 different iterations.  Found a solution that moves 919 regions; Going from a computed imbalance of 0.014044032500016422 to a new imbalance of 6.013394422172574E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.10211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:17,957 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv221933925:81 , srv1568068485:82 , srv385959515:82 , srv2041027691:83 , srv258178445:83 , srv515961396:83 , srv1411834182:84 , srv1608298365:84 , srv170133179:84 , srv1811824875:84 , srv2063301828:84 , srv281911471:84 , srv282935901:84 , srv297458423:84 , srv328021826:84 , srv615894080:84 , srv717408535:84 , srv778156912:84 , srv873238380:84 , srv879517303:84 , srv1039944822:85 , srv1093488197:85 , srv1123308605:85 , srv1133656394:85 , srv1292145949:85 , srv1298388028:85 , srv133175769:85 , srv136939393:85 , srv137667465:85 , srv1419785319:85 , srv1490330925:85 , srv1536927478:85 , srv1573704359:85 , srv1651458744:85 , srv1653424303:85 , srv16558298:85 , srv1761498848:85 , srv1767582992:85 , srv1801515376:85 , srv1883862821:85 , srv2021207184:85 , srv249030372:85 , srv252592563:85 , srv365084930:85 , srv403866941:85 , srv506095823:85 , srv666640949:85 , srv856598904:85 , srv907185366:85 , srv923504960:85 , srv101051400:86 , srv1022143361:86 , srv1045393640:86 , srv1078921953:86 , srv1091230119:86 , srv115252029:86 , srv1161100913:86 , srv1170923191:86 , srv1227236776:86 , srv1649249172:86 , srv1746303627:86 , srv1757453936:86 , srv1895006385:86 , srv1908426365:86 , srv1938366169:86 , srv1996352230:86 , srv2054578763:86 , srv2057505091:86 , srv2140307477:86 , srv300409840:86 , srv347782304:86 , srv367589924:86 , srv402003910:86 , srv435557300:86 , srv633786773:86 , srv63403952:86 , srv680801522:86 , srv752825282:86 , srv827237230:86 , srv894257379:86 , srv897500745:86 , srv1201490298:87 , srv1267386458:87 , srv1313082732:87 , srv13474807:87 , srv1730205604:87 , srv303217466:87 , srv762262772:87 , srv906146083:87 , srv1083896039:88 , srv1875190880:88 , srv1936529187:88 , srv2101252202:88 , srv448501426:88 , srv498049780:88 , srv533290141:88 , srv659715862:88 , srv723660757:88 , srv97260617:89 , srv1501349498:537 }
2024-01-03 07:53:17,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:17,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1292145949=16, srv873238380=92, srv1083896039=5, srv1039944822=2, srv723660757=86, srv1490330925=25, srv762262772=88, srv1123308605=8, srv1746303627=37, srv1501349498=26, srv282935901=62, srv367589924=69, srv435557300=73, srv778156912=89, srv1608298365=30, srv63403952=81, srv365084930=68, srv1811824875=42, srv1895006385=45, srv249030372=58, srv1161100913=11, srv1757453936=38, srv1201490298=13, srv1883862821=44, srv136939393=21, srv856598904=91, srv403866941=72, srv615894080=79, srv659715862=82, srv1653424303=33, srv2041027691=51, srv221933925=57, srv2063301828=54, srv498049780=75, srv907185366=97, srv1938366169=48, srv515961396=77, srv717408535=85, srv533290141=78, srv97260617=99, srv1045393640=3, srv2057505091=53, srv1313082732=18, srv1093488197=7, srv1649249172=31, srv752825282=87, srv923504960=98, srv1573704359=29, srv1227236776=14, srv133175769=19, srv300409840=64, srv633786773=80, srv16558298=34, srv101051400=0, srv281911471=61, srv1419785319=24, srv906146083=96, srv1767582992=40, srv347782304=67, srv1022143361=1, srv402003910=71, srv897500745=95, srv2021207184=50, srv1730205604=36, srv879517303=93, srv1091230119=6, srv303217466=65, srv1875190880=43, srv1536927478=27, srv115252029=10, srv328021826=66, srv258178445=60, srv137667465=22, srv170133179=35, srv297458423=63, srv252592563=59, srv506095823=76, srv1078921953=4, srv448501426=74, srv894257379=94, srv2140307477=56, srv1170923191=12, srv2101252202=55, srv1761498848=39, srv1996352230=49, srv1936529187=47, srv1411834182=23, srv680801522=84, srv1801515376=41, srv1298388028=17, srv385959515=70, srv1133656394=9, srv1267386458=15, srv1651458744=32, srv13474807=20, srv1568068485=28, srv2054578763=52, srv1908426365=46, srv666640949=83, srv827237230=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:17,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 07:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 07:53:17,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 07:53:17,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 07:53:17,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 07:53:17,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 07:53:17,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 07:53:18,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:18,089 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 07:53:18,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.949044666797674E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005540945998106677); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005540945998106677); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 07:53:18,419 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations.  Found a solution that moves 214 regions; Going from a computed imbalance of 5.949044666797674E-4 to a new imbalance of 9.94339286372956E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.023777777777777776); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:18,421 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1757453936:83 , srv1936529187:83 , srv2101252202:83 , srv435557300:83 , srv448501426:83 , srv659715862:83 , srv723660757:83 , srv1022143361:84 , srv1083896039:84 , srv1123308605:84 , srv1161100913:84 , srv1201490298:84 , srv1267386458:84 , srv1292145949:84 , srv13474807:84 , srv1411834182:84 , srv1419785319:84 , srv1536927478:84 , srv1568068485:84 , srv1649249172:84 , srv16558298:84 , srv1761498848:84 , srv1811824875:84 , srv1875190880:84 , srv1908426365:84 , srv1938366169:84 , srv1996352230:84 , srv2021207184:84 , srv2140307477:84 , srv252592563:84 , srv258178445:84 , srv300409840:84 , srv328021826:84 , srv367589924:84 , srv498049780:84 , srv615894080:84 , srv63403952:84 , srv680801522:84 , srv717408535:84 , srv762262772:84 , srv827237230:84 , srv856598904:84 , srv873238380:84 , srv894257379:84 , srv897500745:84 , srv97260617:84 , srv101051400:85 , srv1039944822:85 , srv1045393640:85 , srv1078921953:85 , srv1091230119:85 , srv1093488197:85 , srv1133656394:85 , srv115252029:85 , srv1170923191:85 , srv1227236776:85 , srv1298388028:85 , srv1313082732:85 , srv133175769:85 , srv136939393:85 , srv137667465:85 , srv1490330925:85 , srv1573704359:85 , srv1608298365:85 , srv1651458744:85 , srv1653424303:85 , srv170133179:85 , srv1730205604:85 , srv1746303627:85 , srv1767582992:85 , srv1801515376:85 , srv1883862821:85 , srv1895006385:85 , srv2041027691:85 , srv2054578763:85 , srv2057505091:85 , srv2063301828:85 , srv221933925:85 , srv281911471:85 , srv282935901:85 , srv297458423:85 , srv303217466:85 , srv347782304:85 , srv365084930:85 , srv385959515:85 , srv402003910:85 , srv403866941:85 , srv506095823:85 , srv515961396:85 , srv533290141:85 , srv633786773:85 , srv666640949:85 , srv752825282:85 , srv879517303:85 , srv906146083:85 , srv907185366:85 , srv249030372:86 , srv778156912:86 , srv923504960:86 , srv1501349498:635 }
2024-01-03 07:53:18,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1292145949=16, srv873238380=92, srv1083896039=5, srv1039944822=2, srv723660757=86, srv1490330925=25, srv762262772=88, srv1123308605=8, srv1746303627=37, srv1501349498=26, srv282935901=62, srv367589924=69, srv435557300=73, srv778156912=89, srv1608298365=30, srv63403952=81, srv365084930=68, srv1811824875=42, srv1895006385=45, srv249030372=58, srv1161100913=11, srv1757453936=38, srv1201490298=13, srv1883862821=44, srv136939393=21, srv856598904=91, srv403866941=72, srv615894080=79, srv659715862=82, srv1653424303=33, srv2041027691=51, srv221933925=57, srv2063301828=54, srv498049780=75, srv907185366=97, srv1938366169=48, srv515961396=77, srv717408535=85, srv533290141=78, srv97260617=99, srv1045393640=3, srv2057505091=53, srv1313082732=18, srv1093488197=7, srv1649249172=31, srv752825282=87, srv923504960=98, srv1573704359=29, srv1227236776=14, srv133175769=19, srv300409840=64, srv633786773=80, srv16558298=34, srv101051400=0, srv281911471=61, srv1419785319=24, srv906146083=96, srv1767582992=40, srv347782304=67, srv1022143361=1, srv402003910=71, srv897500745=95, srv2021207184=50, srv1730205604=36, srv879517303=93, srv1091230119=6, srv303217466=65, srv1875190880=43, srv1536927478=27, srv115252029=10, srv328021826=66, srv258178445=60, srv137667465=22, srv170133179=35, srv297458423=63, srv252592563=59, srv506095823=76, srv1078921953=4, srv448501426=74, srv894257379=94, srv2140307477=56, srv1170923191=12, srv2101252202=55, srv1761498848=39, srv1996352230=49, srv1936529187=47, srv1411834182=23, srv680801522=84, srv1801515376=41, srv1298388028=17, srv385959515=70, srv1133656394=9, srv1267386458=15, srv1651458744=32, srv13474807=20, srv1568068485=28, srv2054578763=52, srv1908426365=46, srv666640949=83, srv827237230=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 07:53:18,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 07:53:18,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 07:53:18,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 07:53:18,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 07:53:18,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 07:53:18,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 07:53:18,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 07:53:18,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 07:53:18,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 07:53:18,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 07:53:18,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 07:53:18,563 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.4449328365012E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:18,564 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1757453936:83 , srv1936529187:83 , srv2101252202:83 , srv435557300:83 , srv448501426:83 , srv659715862:83 , srv723660757:83 , srv1022143361:84 , srv1083896039:84 , srv1123308605:84 , srv1161100913:84 , srv1201490298:84 , srv1267386458:84 , srv1292145949:84 , srv13474807:84 , srv1411834182:84 , srv1419785319:84 , srv1536927478:84 , srv1568068485:84 , srv1649249172:84 , srv16558298:84 , srv1761498848:84 , srv1811824875:84 , srv1875190880:84 , srv1908426365:84 , srv1938366169:84 , srv1996352230:84 , srv2021207184:84 , srv2140307477:84 , srv252592563:84 , srv258178445:84 , srv300409840:84 , srv328021826:84 , srv367589924:84 , srv498049780:84 , srv615894080:84 , srv63403952:84 , srv680801522:84 , srv717408535:84 , srv762262772:84 , srv827237230:84 , srv856598904:84 , srv873238380:84 , srv894257379:84 , srv897500745:84 , srv97260617:84 , srv101051400:85 , srv1039944822:85 , srv1045393640:85 , srv1078921953:85 , srv1091230119:85 , srv1093488197:85 , srv1133656394:85 , srv115252029:85 , srv1170923191:85 , srv1227236776:85 , srv1298388028:85 , srv1313082732:85 , srv133175769:85 , srv136939393:85 , srv137667465:85 , srv1490330925:85 , srv1573704359:85 , srv1608298365:85 , srv1651458744:85 , srv1653424303:85 , srv170133179:85 , srv1730205604:85 , srv1746303627:85 , srv1767582992:85 , srv1801515376:85 , srv1883862821:85 , srv1895006385:85 , srv2041027691:85 , srv2054578763:85 , srv2057505091:85 , srv2063301828:85 , srv221933925:85 , srv281911471:85 , srv282935901:85 , srv297458423:85 , srv303217466:85 , srv347782304:85 , srv365084930:85 , srv385959515:85 , srv402003910:85 , srv403866941:85 , srv506095823:85 , srv515961396:85 , srv533290141:85 , srv633786773:85 , srv666640949:85 , srv752825282:85 , srv879517303:85 , srv906146083:85 , srv907185366:85 , srv249030372:86 , srv778156912:86 , srv923504960:86 , srv1501349498:635 }
2024-01-03 07:53:18,595 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=292 (was 274) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=3179 (was 3762)
2024-01-03 07:53:18,603 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=292, ProcessCount=167, AvailableMemoryMB=3178
2024-01-03 07:53:18,604 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 07:53:18,604 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-03 07:53:18,619 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102921256:84 , srv1162417963:84 , srv117655783:84 , srv1253093406:84 , srv1262155127:84 , srv1458712096:84 , srv1467281472:84 , srv1518150373:84 , srv164585621:84 , srv1648384746:84 , srv1713775040:84 , srv1734148460:84 , srv1911324195:84 , srv1950738851:84 , srv197719062:84 , srv2047628188:84 , srv2118449319:84 , srv28879825:84 , srv392867239:84 , srv490430296:84 , srv491927241:84 , srv553314229:84 , srv645599398:84 , srv670440014:84 , srv814196919:84 , srv881258660:84 , srv923623250:84 , srv936111861:84 , srv94976654:84 , srv1984399474:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-03 07:53:18,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:53:18,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1984399474=15, srv1458712096=5, srv490430296=20, srv1253093406=3, srv1950738851=13, srv936111861=28, srv94976654=29, srv923623250=27, srv117655783=2, srv1734148460=11, srv1911324195=12, srv102921256=0, srv1467281472=6, srv28879825=18, srv491927241=21, srv1713775040=10, srv164585621=8, srv2118449319=17, srv1648384746=9, srv1162417963=1, srv553314229=22, srv392867239=19, srv2047628188=16, srv670440014=24, srv1518150373=7, srv881258660=26, srv814196919=25, srv645599398=23, srv197719062=14, srv1262155127=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-03 07:53:18,663 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1109787710273522, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:53:18,663 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912542733747997, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1109787710273522, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-03 07:55:18,642 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6149745 different iterations.  Found a solution that moves 1844 regions; Going from a computed imbalance of 0.9912542733747997 to a new imbalance of 4.303987990979898E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.682962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:55:18,648 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv102921256:90 , srv1162417963:90 , srv117655783:90 , srv1253093406:90 , srv1262155127:90 , srv1458712096:90 , srv1467281472:90 , srv1518150373:90 , srv164585621:90 , srv1648384746:90 , srv1713775040:90 , srv1734148460:90 , srv1911324195:90 , srv1950738851:90 , srv197719062:90 , srv1984399474:90 , srv2047628188:90 , srv2118449319:90 , srv28879825:90 , srv392867239:90 , srv490430296:90 , srv491927241:90 , srv553314229:90 , srv645599398:90 , srv670440014:90 , srv814196919:90 , srv881258660:90 , srv923623250:90 , srv936111861:90 , srv94976654:90 }
2024-01-03 07:55:18,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 07:55:18,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1984399474=15, srv1458712096=5, srv490430296=20, srv1253093406=3, srv1950738851=13, srv936111861=28, srv94976654=29, srv923623250=27, srv117655783=2, srv1734148460=11, srv1911324195=12, srv102921256=0, srv1467281472=6, srv28879825=18, srv491927241=21, srv1713775040=10, srv164585621=8, srv2118449319=17, srv1648384746=9, srv1162417963=1, srv553314229=22, srv392867239=19, srv2047628188=16, srv670440014=24, srv1518150373=7, srv881258660=26, srv814196919=25, srv645599398=23, srv197719062=14, srv1262155127=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 07:55:18,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 07:55:18,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 07:55:18,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 07:55:18,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-03 07:55:18,667 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 07:55:18,667 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv102921256:90 , srv1162417963:90 , srv117655783:90 , srv1253093406:90 , srv1262155127:90 , srv1458712096:90 , srv1467281472:90 , srv1518150373:90 , srv164585621:90 , srv1648384746:90 , srv1713775040:90 , srv1734148460:90 , srv1911324195:90 , srv1950738851:90 , srv197719062:90 , srv1984399474:90 , srv2047628188:90 , srv2118449319:90 , srv28879825:90 , srv392867239:90 , srv490430296:90 , srv491927241:90 , srv553314229:90 , srv645599398:90 , srv670440014:90 , srv814196919:90 , srv881258660:90 , srv923623250:90 , srv936111861:90 , srv94976654:90 }
2024-01-03 07:55:18,682 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=233 (was 292), ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=2771 (was 3178)