2023-09-24 02:56:51,233 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-24 02:56:51,466 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-24 02:56:51,551 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-24 02:56:51,601 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-24 02:56:51,607 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-24 02:56:51,619 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=107, ProcessCount=171, AvailableMemoryMB=7659
2023-09-24 02:56:51,620 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-24 02:56:51,621 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-24 02:56:51,749 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000741234:84 , srv1023352036:84 , srv105761790:84 , srv1059902263:84 , srv1074756139:84 , srv1116352293:84 , srv1130344593:84 , srv1138340454:84 , srv1144481209:84 , srv1145468130:84 , srv116551713:84 , srv1180782686:84 , srv1206994934:84 , srv1233176407:84 , srv1293724850:84 , srv1303993509:84 , srv132890329:84 , srv136663077:84 , srv1368218451:84 , srv1410434027:84 , srv1427688967:84 , srv1433997317:84 , srv1448216340:84 , srv1471812182:84 , srv1472554123:84 , srv1529247826:84 , srv1535902771:84 , srv1536437362:84 , srv1537699329:84 , srv1538203128:84 , srv1539093435:84 , srv154258946:84 , srv1546430020:84 , srv1556020451:84 , srv1599812473:84 , srv160656740:84 , srv1607755078:84 , srv1608648499:84 , srv162715713:84 , srv1638519126:84 , srv168646656:84 , srv1753632339:84 , srv1780617624:84 , srv178068315:84 , srv1792488764:84 , srv1828540491:84 , srv1873683005:84 , srv192528426:84 , srv1950768726:84 , srv1988501353:84 , srv2008836853:84 , srv2020541526:84 , srv2027351274:84 , srv2085339456:84 , srv2101529470:84 , srv2127166749:84 , srv2139597318:84 , srv219818890:84 , srv242380032:84 , srv252393132:84 , srv256701952:84 , srv318755964:84 , srv357853844:84 , srv367221536:84 , srv396512818:84 , srv435324371:84 , srv43759770:84 , srv450471410:84 , srv495466616:84 , srv520455608:84 , srv552991342:84 , srv582845252:84 , srv595572651:84 , srv598407526:84 , srv621460875:84 , srv633375640:84 , srv658454315:84 , srv688378372:84 , srv688957509:84 , srv698630290:84 , srv719464274:84 , srv720284268:84 , srv788406422:84 , srv789644633:84 , srv79179422:84 , srv79327529:84 , srv796480203:84 , srv797942006:84 , srv801329435:84 , srv802205440:84 , srv826834844:84 , srv907174108:84 , srv924523118:84 , srv935674151:84 , srv956932972:84 , srv958610751:84 , srv984922829:84 , srv992962362:84 , srv993586367:84 , srv2005478772:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-24 02:56:51,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:51,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv958610751=96, srv1180782686=11, srv396512818=65, srv633375640=76, srv1546430020=32, srv2027351274=53, srv797942006=88, srv1448216340=22, srv984922829=97, srv1792488764=44, srv105761790=2, srv688378372=78, srv154258946=31, srv192528426=47, srv252393132=60, srv1472554123=24, srv318755964=62, srv720284268=82, srv1138340454=7, srv1427688967=20, srv621460875=75, srv162715713=38, srv2008836853=51, srv79179422=85, srv1116352293=5, srv219818890=58, srv993586367=99, srv796480203=87, srv1433997317=21, srv788406422=83, srv116551713=10, srv2127166749=56, srv789644633=84, srv1556020451=33, srv79327529=86, srv801329435=89, srv2101529470=55, srv2085339456=54, srv1233176407=13, srv956932972=95, srv1145468130=9, srv132890329=16, srv1780617624=42, srv907174108=92, srv688957509=79, srv1828540491=45, srv1023352036=1, srv1538203128=29, srv1000741234=0, srv1471812182=23, srv552991342=71, srv802205440=90, srv1303993509=15, srv1535902771=26, srv1539093435=30, srv2005478772=50, srv242380032=59, srv935674151=94, srv826834844=91, srv495466616=69, srv2020541526=52, srv2139597318=57, srv367221536=64, srv357853844=63, srv1059902263=3, srv1529247826=25, srv1608648499=37, srv698630290=80, srv1638519126=39, srv1537699329=28, srv1206994934=12, srv1753632339=41, srv1144481209=8, srv1536437362=27, srv595572651=73, srv256701952=61, srv435324371=66, srv1130344593=6, srv598407526=74, srv136663077=17, srv1988501353=49, srv924523118=93, srv1410434027=19, srv1599812473=34, srv520455608=70, srv658454315=77, srv1607755078=36, srv1873683005=46, srv582845252=72, srv168646656=40, srv1293724850=14, srv178068315=43, srv450471410=68, srv1950768726=48, srv719464274=81, srv1074756139=4, srv1368218451=18, srv160656740=35, srv43759770=67, srv992962362=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-24 02:56:51,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-24 02:56:51,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-24 02:56:51,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-24 02:56:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-24 02:56:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-24 02:56:51,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-24 02:56:51,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-24 02:56:52,113 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,113 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-24 02:56:52,113 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-24 02:56:52,343 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 477 ms to try 5000 different iterations.  Found a solution that moves 4748 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17196283082076216. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959232329440089, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414782111947379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5275555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959232329440089, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23491666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,352 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1233176407:82 , srv1599812473:82 , srv1608648499:82 , srv105761790:83 , srv154258946:83 , srv1546430020:83 , srv1988501353:83 , srv43759770:83 , srv788406422:83 , srv79327529:83 , srv924523118:83 , srv935674151:83 , srv1130344593:84 , srv116551713:84 , srv1368218451:84 , srv1427688967:84 , srv1472554123:84 , srv789644633:84 , srv1116352293:85 , srv1180782686:85 , srv1293724850:85 , srv132890329:85 , srv1471812182:85 , srv1537699329:85 , srv1556020451:85 , srv162715713:85 , srv178068315:85 , srv1950768726:85 , srv2101529470:85 , srv318755964:85 , srv367221536:85 , srv396512818:85 , srv450471410:85 , srv495466616:85 , srv658454315:85 , srv801329435:85 , srv802205440:85 , srv958610751:85 , srv1023352036:86 , srv1059902263:86 , srv1074756139:86 , srv1303993509:86 , srv1529247826:86 , srv1638519126:86 , srv2127166749:86 , srv242380032:86 , srv252393132:86 , srv256701952:86 , srv552991342:86 , srv621460875:86 , srv688378372:86 , srv688957509:86 , srv698630290:86 , srv992962362:86 , srv993586367:86 , srv1000741234:87 , srv1206994934:87 , srv1536437362:87 , srv1539093435:87 , srv160656740:87 , srv1607755078:87 , srv1780617624:87 , srv1873683005:87 , srv2027351274:87 , srv582845252:87 , srv598407526:87 , srv796480203:87 , srv956932972:87 , srv1410434027:88 , srv1538203128:88 , srv1753632339:88 , srv2139597318:88 , srv595572651:88 , srv633375640:88 , srv79179422:88 , srv826834844:88 , srv984922829:88 , srv1138340454:89 , srv1144481209:89 , srv136663077:89 , srv192528426:89 , srv2020541526:89 , srv2085339456:89 , srv219818890:89 , srv357853844:89 , srv719464274:89 , srv797942006:89 , srv907174108:89 , srv1145468130:90 , srv1433997317:90 , srv168646656:90 , srv1828540491:90 , srv2008836853:90 , srv720284268:90 , srv1448216340:91 , srv1535902771:91 , srv1792488764:91 , srv435324371:91 , srv520455608:93 , srv2005478772:442 }
2023-09-24 02:56:52,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:52,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv958610751=96, srv1180782686=11, srv396512818=65, srv633375640=76, srv1546430020=32, srv2027351274=53, srv797942006=88, srv1448216340=22, srv984922829=97, srv1792488764=44, srv105761790=2, srv688378372=78, srv154258946=31, srv192528426=47, srv252393132=60, srv1472554123=24, srv318755964=62, srv720284268=82, srv1138340454=7, srv1427688967=20, srv621460875=75, srv162715713=38, srv2008836853=51, srv79179422=85, srv1116352293=5, srv219818890=58, srv993586367=99, srv796480203=87, srv1433997317=21, srv788406422=83, srv116551713=10, srv2127166749=56, srv789644633=84, srv1556020451=33, srv79327529=86, srv801329435=89, srv2101529470=55, srv2085339456=54, srv1233176407=13, srv956932972=95, srv1145468130=9, srv132890329=16, srv1780617624=42, srv907174108=92, srv688957509=79, srv1828540491=45, srv1023352036=1, srv1538203128=29, srv1000741234=0, srv1471812182=23, srv552991342=71, srv802205440=90, srv1303993509=15, srv1535902771=26, srv1539093435=30, srv2005478772=50, srv242380032=59, srv935674151=94, srv826834844=91, srv495466616=69, srv2020541526=52, srv2139597318=57, srv367221536=64, srv357853844=63, srv1059902263=3, srv1529247826=25, srv1608648499=37, srv698630290=80, srv1638519126=39, srv1537699329=28, srv1206994934=12, srv1753632339=41, srv1144481209=8, srv1536437362=27, srv595572651=73, srv256701952=61, srv435324371=66, srv1130344593=6, srv598407526=74, srv136663077=17, srv1988501353=49, srv924523118=93, srv1410434027=19, srv1599812473=34, srv520455608=70, srv658454315=77, srv1607755078=36, srv1873683005=46, srv582845252=72, srv168646656=40, srv1293724850=14, srv178068315=43, srv450471410=68, srv1950768726=48, srv719464274=81, srv1074756139=4, srv1368218451=18, srv160656740=35, srv43759770=67, srv992962362=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:52,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:52,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:52,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:52,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-24 02:56:52,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-24 02:56:52,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-24 02:56:52,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-24 02:56:52,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-24 02:56:52,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-24 02:56:52,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-24 02:56:52,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-24 02:56:52,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959232329440089, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414782111947379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959232329440089, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23491666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,485 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-24 02:56:52,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17192958461417676, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959232329440089, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414782111947379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959232329440089, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23491666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-24 02:56:52,690 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 284 ms to try 5000 different iterations.  Found a solution that moves 2583 regions; Going from a computed imbalance of 0.17192958461417676 to a new imbalance of 0.014532668897476507. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020507952888751298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032529016659032156, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.287, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020507952888751298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,702 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv520455608:79 , srv1433997317:80 , srv136663077:81 , srv1535902771:81 , srv1753632339:81 , srv1828540491:81 , srv2008836853:81 , srv219818890:81 , srv1138340454:82 , srv1448216340:82 , srv168646656:82 , srv1792488764:82 , srv2020541526:82 , srv357853844:82 , srv633375640:82 , srv720284268:82 , srv907174108:82 , srv1023352036:83 , srv2085339456:83 , srv252393132:83 , srv598407526:83 , srv719464274:83 , srv1303993509:84 , srv1410434027:84 , srv1536437362:84 , srv192528426:84 , srv435324371:84 , srv621460875:84 , srv984922829:84 , srv1145468130:85 , srv1529247826:85 , srv1607755078:85 , srv1780617624:85 , srv2139597318:85 , srv595572651:85 , srv788406422:85 , srv789644633:85 , srv796480203:85 , srv826834844:85 , srv993586367:85 , srv1000741234:86 , srv1144481209:86 , srv1180782686:86 , srv1472554123:86 , srv1538203128:86 , srv1539093435:86 , srv162715713:86 , srv2101529470:86 , srv242380032:86 , srv318755964:86 , srv396512818:86 , srv552991342:86 , srv582845252:86 , srv688378372:86 , srv698630290:86 , srv79179422:86 , srv797942006:86 , srv956932972:86 , srv958610751:86 , srv992962362:86 , srv1116352293:87 , srv1130344593:87 , srv1206994934:87 , srv1233176407:87 , srv1293724850:87 , srv1471812182:87 , srv1873683005:87 , srv2127166749:87 , srv256701952:87 , srv801329435:87 , srv802205440:87 , srv1059902263:88 , srv116551713:88 , srv1368218451:88 , srv154258946:88 , srv1546430020:88 , srv1556020451:88 , srv160656740:88 , srv1638519126:88 , srv1988501353:88 , srv2027351274:88 , srv43759770:88 , srv495466616:88 , srv658454315:88 , srv935674151:88 , srv1074756139:89 , srv132890329:89 , srv1427688967:89 , srv1537699329:89 , srv367221536:89 , srv450471410:89 , srv688957509:89 , srv924523118:89 , srv105761790:90 , srv178068315:90 , srv1950768726:90 , srv1599812473:91 , srv79327529:91 , srv1608648499:92 , srv2005478772:514 }
2023-09-24 02:56:52,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:52,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv958610751=96, srv1180782686=11, srv396512818=65, srv633375640=76, srv1546430020=32, srv2027351274=53, srv797942006=88, srv1448216340=22, srv984922829=97, srv1792488764=44, srv105761790=2, srv688378372=78, srv154258946=31, srv192528426=47, srv252393132=60, srv1472554123=24, srv318755964=62, srv720284268=82, srv1138340454=7, srv1427688967=20, srv621460875=75, srv162715713=38, srv2008836853=51, srv79179422=85, srv1116352293=5, srv219818890=58, srv993586367=99, srv796480203=87, srv1433997317=21, srv788406422=83, srv116551713=10, srv2127166749=56, srv789644633=84, srv1556020451=33, srv79327529=86, srv801329435=89, srv2101529470=55, srv2085339456=54, srv1233176407=13, srv956932972=95, srv1145468130=9, srv132890329=16, srv1780617624=42, srv907174108=92, srv688957509=79, srv1828540491=45, srv1023352036=1, srv1538203128=29, srv1000741234=0, srv1471812182=23, srv552991342=71, srv802205440=90, srv1303993509=15, srv1535902771=26, srv1539093435=30, srv2005478772=50, srv242380032=59, srv935674151=94, srv826834844=91, srv495466616=69, srv2020541526=52, srv2139597318=57, srv367221536=64, srv357853844=63, srv1059902263=3, srv1529247826=25, srv1608648499=37, srv698630290=80, srv1638519126=39, srv1537699329=28, srv1206994934=12, srv1753632339=41, srv1144481209=8, srv1536437362=27, srv595572651=73, srv256701952=61, srv435324371=66, srv1130344593=6, srv598407526=74, srv136663077=17, srv1988501353=49, srv924523118=93, srv1410434027=19, srv1599812473=34, srv520455608=70, srv658454315=77, srv1607755078=36, srv1873683005=46, srv582845252=72, srv168646656=40, srv1293724850=14, srv178068315=43, srv450471410=68, srv1950768726=48, srv719464274=81, srv1074756139=4, srv1368218451=18, srv160656740=35, srv43759770=67, srv992962362=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:52,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:52,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-24 02:56:52,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-24 02:56:52,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-24 02:56:52,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-24 02:56:52,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-24 02:56:52,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-24 02:56:52,815 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020507952888751298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032529016659032156, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020507952888751298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,815 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-24 02:56:52,816 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014514582344904868, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020507952888751298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032529016659032156, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020507952888751298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-24 02:56:52,976 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations.  Found a solution that moves 963 regions; Going from a computed imbalance of 0.014514582344904868 to a new imbalance of 5.411728842029864E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005674374722806451); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007654139963827327); MoveCostFunction : (multiplier=7.0, imbalance=0.107, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005674374722806451); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:52,978 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1144481209:82 , srv1145468130:82 , srv105761790:83 , srv1059902263:83 , srv1074756139:83 , srv1206994934:83 , srv132890329:83 , srv1538203128:83 , srv1599812473:83 , srv160656740:83 , srv178068315:83 , srv1873683005:83 , srv2027351274:83 , srv2127166749:83 , srv367221536:83 , srv435324371:83 , srv79327529:83 , srv797942006:83 , srv1448216340:84 , srv1537699329:84 , srv1608648499:84 , srv1638519126:84 , srv1950768726:84 , srv450471410:84 , srv495466616:84 , srv552991342:84 , srv595572651:84 , srv688957509:84 , srv79179422:84 , srv1000741234:85 , srv116551713:85 , srv1368218451:85 , srv1427688967:85 , srv1471812182:85 , srv1539093435:85 , srv1556020451:85 , srv168646656:85 , srv1780617624:85 , srv1792488764:85 , srv192528426:85 , srv2085339456:85 , srv2139597318:85 , srv256701952:85 , srv318755964:85 , srv396512818:85 , srv582845252:85 , srv658454315:85 , srv826834844:85 , srv924523118:85 , srv935674151:85 , srv956932972:85 , srv984922829:85 , srv992962362:85 , srv1116352293:86 , srv1130344593:86 , srv1293724850:86 , srv1410434027:86 , srv1529247826:86 , srv1535902771:86 , srv154258946:86 , srv1546430020:86 , srv1607755078:86 , srv162715713:86 , srv1988501353:86 , srv2008836853:86 , srv242380032:86 , srv43759770:86 , srv520455608:86 , srv598407526:86 , srv688378372:86 , srv698630290:86 , srv719464274:86 , srv802205440:86 , srv907174108:86 , srv993586367:86 , srv1138340454:87 , srv1180782686:87 , srv1433997317:87 , srv1472554123:87 , srv1536437362:87 , srv1828540491:87 , srv2020541526:87 , srv2101529470:87 , srv219818890:87 , srv357853844:87 , srv621460875:87 , srv633375640:87 , srv720284268:87 , srv796480203:87 , srv958610751:87 , srv1233176407:88 , srv136663077:88 , srv1753632339:88 , srv252393132:88 , srv789644633:88 , srv801329435:88 , srv1023352036:89 , srv1303993509:89 , srv788406422:89 , srv2005478772:552 }
2023-09-24 02:56:52,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:52,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv958610751=96, srv1180782686=11, srv396512818=65, srv633375640=76, srv1546430020=32, srv2027351274=53, srv797942006=88, srv1448216340=22, srv984922829=97, srv1792488764=44, srv105761790=2, srv688378372=78, srv154258946=31, srv192528426=47, srv252393132=60, srv1472554123=24, srv318755964=62, srv720284268=82, srv1138340454=7, srv1427688967=20, srv621460875=75, srv162715713=38, srv2008836853=51, srv79179422=85, srv1116352293=5, srv219818890=58, srv993586367=99, srv796480203=87, srv1433997317=21, srv788406422=83, srv116551713=10, srv2127166749=56, srv789644633=84, srv1556020451=33, srv79327529=86, srv801329435=89, srv2101529470=55, srv2085339456=54, srv1233176407=13, srv956932972=95, srv1145468130=9, srv132890329=16, srv1780617624=42, srv907174108=92, srv688957509=79, srv1828540491=45, srv1023352036=1, srv1538203128=29, srv1000741234=0, srv1471812182=23, srv552991342=71, srv802205440=90, srv1303993509=15, srv1535902771=26, srv1539093435=30, srv2005478772=50, srv242380032=59, srv935674151=94, srv826834844=91, srv495466616=69, srv2020541526=52, srv2139597318=57, srv367221536=64, srv357853844=63, srv1059902263=3, srv1529247826=25, srv1608648499=37, srv698630290=80, srv1638519126=39, srv1537699329=28, srv1206994934=12, srv1753632339=41, srv1144481209=8, srv1536437362=27, srv595572651=73, srv256701952=61, srv435324371=66, srv1130344593=6, srv598407526=74, srv136663077=17, srv1988501353=49, srv924523118=93, srv1410434027=19, srv1599812473=34, srv520455608=70, srv658454315=77, srv1607755078=36, srv1873683005=46, srv582845252=72, srv168646656=40, srv1293724850=14, srv178068315=43, srv450471410=68, srv1950768726=48, srv719464274=81, srv1074756139=4, srv1368218451=18, srv160656740=35, srv43759770=67, srv992962362=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:52,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:52,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:52,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:52,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:52,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:52,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-24 02:56:52,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-24 02:56:52,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-24 02:56:52,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-24 02:56:52,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-24 02:56:52,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-24 02:56:52,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-24 02:56:52,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-24 02:56:53,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-24 02:56:53,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-24 02:56:53,077 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005674374722806451); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007654139963827327); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005674374722806451); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:53,077 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-24 02:56:53,078 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.344298140804588E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005674374722806451); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007654139963827327); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005674374722806451); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-24 02:56:53,294 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 286 ms to try 5000 different iterations.  Found a solution that moves 251 regions; Going from a computed imbalance of 5.344298140804588E-4 to a new imbalance of 1.2956114433165202E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.02788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:53,295 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv801329435:82 , srv1303993509:83 , srv1950768726:83 , srv688378372:83 , srv688957509:83 , srv720284268:83 , srv796480203:83 , srv1000741234:84 , srv1023352036:84 , srv1074756139:84 , srv1116352293:84 , srv1138340454:84 , srv1144481209:84 , srv116551713:84 , srv1293724850:84 , srv132890329:84 , srv136663077:84 , srv1410434027:84 , srv1427688967:84 , srv1448216340:84 , srv1471812182:84 , srv1535902771:84 , srv1536437362:84 , srv1537699329:84 , srv1539093435:84 , srv1556020451:84 , srv1607755078:84 , srv1608648499:84 , srv1780617624:84 , srv178068315:84 , srv1792488764:84 , srv1828540491:84 , srv192528426:84 , srv2020541526:84 , srv2027351274:84 , srv2139597318:84 , srv219818890:84 , srv242380032:84 , srv256701952:84 , srv357853844:84 , srv435324371:84 , srv450471410:84 , srv520455608:84 , srv582845252:84 , srv658454315:84 , srv698630290:84 , srv719464274:84 , srv79179422:84 , srv79327529:84 , srv802205440:84 , srv956932972:84 , srv1059902263:85 , srv1130344593:85 , srv1180782686:85 , srv1206994934:85 , srv1233176407:85 , srv1368218451:85 , srv1433997317:85 , srv1472554123:85 , srv1529247826:85 , srv1538203128:85 , srv154258946:85 , srv1546430020:85 , srv160656740:85 , srv162715713:85 , srv1638519126:85 , srv1753632339:85 , srv1873683005:85 , srv1988501353:85 , srv2008836853:85 , srv2085339456:85 , srv2101529470:85 , srv252393132:85 , srv318755964:85 , srv367221536:85 , srv43759770:85 , srv495466616:85 , srv595572651:85 , srv621460875:85 , srv633375640:85 , srv788406422:85 , srv789644633:85 , srv797942006:85 , srv826834844:85 , srv907174108:85 , srv924523118:85 , srv958610751:85 , srv984922829:85 , srv992962362:85 , srv993586367:85 , srv105761790:86 , srv1145468130:86 , srv1599812473:86 , srv168646656:86 , srv2127166749:86 , srv396512818:86 , srv598407526:86 , srv935674151:86 , srv552991342:87 , srv2005478772:634 }
2023-09-24 02:56:53,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:53,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv958610751=96, srv1180782686=11, srv396512818=65, srv633375640=76, srv1546430020=32, srv2027351274=53, srv797942006=88, srv1448216340=22, srv984922829=97, srv1792488764=44, srv105761790=2, srv688378372=78, srv154258946=31, srv192528426=47, srv252393132=60, srv1472554123=24, srv318755964=62, srv720284268=82, srv1138340454=7, srv1427688967=20, srv621460875=75, srv162715713=38, srv2008836853=51, srv79179422=85, srv1116352293=5, srv219818890=58, srv993586367=99, srv796480203=87, srv1433997317=21, srv788406422=83, srv116551713=10, srv2127166749=56, srv789644633=84, srv1556020451=33, srv79327529=86, srv801329435=89, srv2101529470=55, srv2085339456=54, srv1233176407=13, srv956932972=95, srv1145468130=9, srv132890329=16, srv1780617624=42, srv907174108=92, srv688957509=79, srv1828540491=45, srv1023352036=1, srv1538203128=29, srv1000741234=0, srv1471812182=23, srv552991342=71, srv802205440=90, srv1303993509=15, srv1535902771=26, srv1539093435=30, srv2005478772=50, srv242380032=59, srv935674151=94, srv826834844=91, srv495466616=69, srv2020541526=52, srv2139597318=57, srv367221536=64, srv357853844=63, srv1059902263=3, srv1529247826=25, srv1608648499=37, srv698630290=80, srv1638519126=39, srv1537699329=28, srv1206994934=12, srv1753632339=41, srv1144481209=8, srv1536437362=27, srv595572651=73, srv256701952=61, srv435324371=66, srv1130344593=6, srv598407526=74, srv136663077=17, srv1988501353=49, srv924523118=93, srv1410434027=19, srv1599812473=34, srv520455608=70, srv658454315=77, srv1607755078=36, srv1873683005=46, srv582845252=72, srv168646656=40, srv1293724850=14, srv178068315=43, srv450471410=68, srv1950768726=48, srv719464274=81, srv1074756139=4, srv1368218451=18, srv160656740=35, srv43759770=67, srv992962362=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:53,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:53,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:53,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:53,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-24 02:56:53,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-24 02:56:53,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-24 02:56:53,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-24 02:56:53,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-24 02:56:53,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-24 02:56:53,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-24 02:56:53,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-24 02:56:53,393 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1198574868518857E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:53,393 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv801329435:82 , srv1303993509:83 , srv1950768726:83 , srv688378372:83 , srv688957509:83 , srv720284268:83 , srv796480203:83 , srv1000741234:84 , srv1023352036:84 , srv1074756139:84 , srv1116352293:84 , srv1138340454:84 , srv1144481209:84 , srv116551713:84 , srv1293724850:84 , srv132890329:84 , srv136663077:84 , srv1410434027:84 , srv1427688967:84 , srv1448216340:84 , srv1471812182:84 , srv1535902771:84 , srv1536437362:84 , srv1537699329:84 , srv1539093435:84 , srv1556020451:84 , srv1607755078:84 , srv1608648499:84 , srv1780617624:84 , srv178068315:84 , srv1792488764:84 , srv1828540491:84 , srv192528426:84 , srv2020541526:84 , srv2027351274:84 , srv2139597318:84 , srv219818890:84 , srv242380032:84 , srv256701952:84 , srv357853844:84 , srv435324371:84 , srv450471410:84 , srv520455608:84 , srv582845252:84 , srv658454315:84 , srv698630290:84 , srv719464274:84 , srv79179422:84 , srv79327529:84 , srv802205440:84 , srv956932972:84 , srv1059902263:85 , srv1130344593:85 , srv1180782686:85 , srv1206994934:85 , srv1233176407:85 , srv1368218451:85 , srv1433997317:85 , srv1472554123:85 , srv1529247826:85 , srv1538203128:85 , srv154258946:85 , srv1546430020:85 , srv160656740:85 , srv162715713:85 , srv1638519126:85 , srv1753632339:85 , srv1873683005:85 , srv1988501353:85 , srv2008836853:85 , srv2085339456:85 , srv2101529470:85 , srv252393132:85 , srv318755964:85 , srv367221536:85 , srv43759770:85 , srv495466616:85 , srv595572651:85 , srv621460875:85 , srv633375640:85 , srv788406422:85 , srv789644633:85 , srv797942006:85 , srv826834844:85 , srv907174108:85 , srv924523118:85 , srv958610751:85 , srv984922829:85 , srv992962362:85 , srv993586367:85 , srv105761790:86 , srv1145468130:86 , srv1599812473:86 , srv168646656:86 , srv2127166749:86 , srv396512818:86 , srv598407526:86 , srv935674151:86 , srv552991342:87 , srv2005478772:634 }
2023-09-24 02:56:53,414 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=107 (was 107), ProcessCount=171 (was 171), AvailableMemoryMB=7184 (was 7659)
2023-09-24 02:56:53,421 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=107, ProcessCount=171, AvailableMemoryMB=7184
2023-09-24 02:56:53,421 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-24 02:56:53,421 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-24 02:56:53,430 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1047591081:84 , srv1075118707:84 , srv110841992:84 , srv1127073242:84 , srv1442795746:84 , srv1498749304:84 , srv1533892231:84 , srv1705951114:84 , srv1759632249:84 , srv1778966447:84 , srv1864000116:84 , srv1884122001:84 , srv1980459576:84 , srv1999113018:84 , srv2063128034:84 , srv2076729446:84 , srv231854430:84 , srv294234923:84 , srv306857912:84 , srv332316433:84 , srv438661296:84 , srv470370637:84 , srv479106681:84 , srv614504519:84 , srv651672941:84 , srv707649598:84 , srv714610863:84 , srv744890462:84 , srv889247307:84 , srv707864171:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-24 02:56:53,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:56:53,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv294234923=17, srv1127073242=3, srv889247307=29, srv1533892231=6, srv1980459576=12, srv2063128034=14, srv1705951114=7, srv1864000116=10, srv231854430=16, srv1759632249=8, srv470370637=21, srv438661296=20, srv110841992=2, srv1999113018=13, srv306857912=18, srv744890462=28, srv1498749304=5, srv1884122001=11, srv707649598=25, srv1075118707=1, srv651672941=24, srv1778966447=9, srv332316433=19, srv1047591081=0, srv479106681=22, srv2076729446=15, srv1442795746=4, srv714610863=27, srv707864171=26, srv614504519=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:56:53,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:56:53,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-24 02:56:53,454 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1554853396778706, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:56:53,454 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991268297249254, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1554853396778706, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-24 02:58:53,444 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9893684 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.991268297249254 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:58:53,450 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1047591081:90 , srv1075118707:90 , srv110841992:90 , srv1127073242:90 , srv1442795746:90 , srv1498749304:90 , srv1533892231:90 , srv1705951114:90 , srv1759632249:90 , srv1778966447:90 , srv1864000116:90 , srv1884122001:90 , srv1980459576:90 , srv1999113018:90 , srv2063128034:90 , srv2076729446:90 , srv231854430:90 , srv294234923:90 , srv306857912:90 , srv332316433:90 , srv438661296:90 , srv470370637:90 , srv479106681:90 , srv614504519:90 , srv651672941:90 , srv707649598:90 , srv707864171:90 , srv714610863:90 , srv744890462:90 , srv889247307:90 }
2023-09-24 02:58:53,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-24 02:58:53,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv294234923=17, srv1127073242=3, srv889247307=29, srv1533892231=6, srv1980459576=12, srv2063128034=14, srv1705951114=7, srv1864000116=10, srv231854430=16, srv1759632249=8, srv470370637=21, srv438661296=20, srv110841992=2, srv1999113018=13, srv306857912=18, srv744890462=28, srv1498749304=5, srv1884122001=11, srv707649598=25, srv1075118707=1, srv651672941=24, srv1778966447=9, srv332316433=19, srv1047591081=0, srv479106681=22, srv2076729446=15, srv1442795746=4, srv714610863=27, srv707864171=26, srv614504519=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-24 02:58:53,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-24 02:58:53,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-24 02:58:53,469 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-24 02:58:53,469 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1047591081:90 , srv1075118707:90 , srv110841992:90 , srv1127073242:90 , srv1442795746:90 , srv1498749304:90 , srv1533892231:90 , srv1705951114:90 , srv1759632249:90 , srv1778966447:90 , srv1864000116:90 , srv1884122001:90 , srv1980459576:90 , srv1999113018:90 , srv2063128034:90 , srv2076729446:90 , srv231854430:90 , srv294234923:90 , srv306857912:90 , srv332316433:90 , srv438661296:90 , srv470370637:90 , srv479106681:90 , srv614504519:90 , srv651672941:90 , srv707649598:90 , srv707864171:90 , srv714610863:90 , srv744890462:90 , srv889247307:90 }
2023-09-24 02:58:53,481 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=113 (was 107) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=8223 (was 7184) - AvailableMemoryMB LEAK? -