2023-09-23 11:53:13,216 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-23 11:53:13,551 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-23 11:53:13,661 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-23 11:53:13,720 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-23 11:53:13,731 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-23 11:53:13,747 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=259, ProcessCount=172, AvailableMemoryMB=3117
2023-09-23 11:53:13,750 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-23 11:53:13,750 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-23 11:53:13,912 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1041039980:84 , srv1093099248:84 , srv1103506571:84 , srv1118398625:84 , srv1119190866:84 , srv1120332450:84 , srv1132763457:84 , srv1134444174:84 , srv1147052269:84 , srv1149272656:84 , srv1185392749:84 , srv1187077157:84 , srv1209647324:84 , srv1210003051:84 , srv1252358455:84 , srv1268353618:84 , srv1276124647:84 , srv1311143374:84 , srv1313651546:84 , srv1316328634:84 , srv1342143217:84 , srv1346733897:84 , srv1351541090:84 , srv1410438513:84 , srv1421929733:84 , srv1446815809:84 , srv1449379238:84 , srv1461508801:84 , srv1602985424:84 , srv1605379565:84 , srv1625798352:84 , srv16510149:84 , srv1668025679:84 , srv168000313:84 , srv1691460081:84 , srv1704129133:84 , srv1705413790:84 , srv1820266700:84 , srv1843684053:84 , srv1846067276:84 , srv1852114209:84 , srv185470342:84 , srv1858758779:84 , srv1877797389:84 , srv1911678238:84 , srv1946947354:84 , srv1990996863:84 , srv2028222241:84 , srv2063704772:84 , srv2074378551:84 , srv2085498089:84 , srv2093115318:84 , srv2104926888:84 , srv2114148340:84 , srv2139247535:84 , srv227322633:84 , srv246274323:84 , srv280632188:84 , srv280845145:84 , srv28995972:84 , srv295724291:84 , srv3087209:84 , srv311338758:84 , srv327164324:84 , srv330380609:84 , srv349477235:84 , srv354199743:84 , srv386706315:84 , srv39830031:84 , srv418157925:84 , srv452255616:84 , srv471799143:84 , srv519655418:84 , srv534436617:84 , srv548635289:84 , srv548782692:84 , srv556862115:84 , srv581928158:84 , srv586355411:84 , srv586488169:84 , srv592738608:84 , srv594415309:84 , srv596589458:84 , srv622975529:84 , srv626834530:84 , srv642567785:84 , srv679219198:84 , srv720834968:84 , srv725845781:84 , srv730230343:84 , srv757810445:84 , srv812286763:84 , srv829302628:84 , srv835838702:84 , srv880097834:84 , srv894489445:84 , srv896279324:84 , srv911253542:84 , srv916784157:84 , srv1757607884:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-23 11:53:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:13,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852114209=41, srv594415309=82, srv327164324=64, srv1605379565=29, srv227322633=56, srv812286763=92, srv1820266700=38, srv452255616=71, srv894489445=96, srv1209647324=12, srv1602985424=28, srv386706315=68, srv471799143=72, srv1118398625=3, srv622975529=84, srv1276124647=16, srv2028222241=48, srv1990996863=47, srv1843684053=39, srv1691460081=34, srv730230343=90, srv1185392749=10, srv39830031=69, srv1149272656=9, srv642567785=86, srv1446815809=25, srv1449379238=26, srv1946947354=46, srv311338758=63, srv1252358455=14, srv185470342=42, srv1210003051=13, srv1134444174=7, srv534436617=74, srv1041039980=0, srv1120332450=5, srv581928158=78, srv592738608=81, srv1410438513=23, srv1346733897=21, srv911253542=98, srv596589458=83, srv2139247535=55, srv725845781=89, srv1268353618=15, srv1421929733=24, srv330380609=65, srv586355411=79, srv246274323=57, srv3087209=62, srv1911678238=45, srv896279324=97, srv1858758779=43, srv1625798352=30, srv1147052269=8, srv2093115318=52, srv626834530=85, srv28995972=60, srv2074378551=50, srv1311143374=17, srv1705413790=36, srv280632188=58, srv1668025679=32, srv354199743=67, srv349477235=66, srv1877797389=44, srv916784157=99, srv548782692=76, srv1351541090=22, srv880097834=95, srv720834968=88, srv418157925=70, srv2104926888=53, srv1093099248=1, srv519655418=73, srv1103506571=2, srv2085498089=51, srv679219198=87, srv835838702=94, srv1757607884=37, srv1132763457=6, srv1119190866=4, srv1313651546=18, srv1846067276=40, srv1187077157=11, srv556862115=77, srv168000313=33, srv757810445=91, srv16510149=31, srv1316328634=19, srv280845145=59, srv1704129133=35, srv1342143217=20, srv1461508801=27, srv2063704772=49, srv548635289=75, srv295724291=61, srv2114148340=54, srv829302628=93, srv586488169=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:14,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-23 11:53:14,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-23 11:53:14,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-23 11:53:14,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-23 11:53:14,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-23 11:53:14,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-23 11:53:14,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-23 11:53:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-23 11:53:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-23 11:53:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-23 11:53:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-23 11:53:14,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-23 11:53:14,210 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:14,210 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-23 11:53:14,210 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-23 11:53:14,551 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 507 ms to try 5000 different iterations.  Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18028866631858495. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05429271787602717, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:14,575 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv534436617:81 , srv1268353618:82 , srv1118398625:83 , srv1134444174:83 , srv1185392749:83 , srv1704129133:83 , srv2074378551:83 , srv28995972:83 , srv327164324:83 , srv581928158:83 , srv1103506571:84 , srv1120332450:84 , srv1147052269:84 , srv1691460081:84 , srv185470342:84 , srv2028222241:84 , srv2093115318:84 , srv548635289:84 , srv757810445:84 , srv911253542:84 , srv1093099248:85 , srv1209647324:85 , srv1313651546:85 , srv1852114209:85 , srv1877797389:85 , srv1946947354:85 , srv227322633:85 , srv246274323:85 , srv3087209:85 , srv386706315:85 , srv471799143:85 , srv586488169:85 , srv622975529:85 , srv720834968:85 , srv730230343:85 , srv880097834:85 , srv916784157:85 , srv1132763457:86 , srv1342143217:86 , srv1421929733:86 , srv16510149:86 , srv1705413790:86 , srv1990996863:86 , srv2114148340:86 , srv2139247535:86 , srv349477235:86 , srv418157925:86 , srv556862115:86 , srv586355411:86 , srv642567785:86 , srv679219198:86 , srv1149272656:87 , srv1210003051:87 , srv1311143374:87 , srv1410438513:87 , srv1449379238:87 , srv1461508801:87 , srv1820266700:87 , srv1846067276:87 , srv2085498089:87 , srv280845145:87 , srv39830031:87 , srv592738608:87 , srv596589458:87 , srv626834530:87 , srv894489445:87 , srv1119190866:88 , srv1346733897:88 , srv1446815809:88 , srv1668025679:88 , srv168000313:88 , srv1843684053:88 , srv1858758779:88 , srv2063704772:88 , srv330380609:88 , srv354199743:88 , srv519655418:88 , srv594415309:88 , srv829302628:88 , srv835838702:88 , srv1187077157:89 , srv1276124647:89 , srv1316328634:89 , srv1351541090:89 , srv280632188:89 , srv548782692:89 , srv896279324:89 , srv1252358455:90 , srv1625798352:90 , srv2104926888:90 , srv311338758:90 , srv812286763:90 , srv1602985424:91 , srv295724291:91 , srv1911678238:92 , srv1041039980:94 , srv1605379565:94 , srv452255616:94 , srv725845781:94 , srv1757607884:424 }
2023-09-23 11:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852114209=41, srv594415309=82, srv327164324=64, srv1605379565=29, srv227322633=56, srv812286763=92, srv1820266700=38, srv452255616=71, srv894489445=96, srv1209647324=12, srv1602985424=28, srv386706315=68, srv471799143=72, srv1118398625=3, srv622975529=84, srv1276124647=16, srv2028222241=48, srv1990996863=47, srv1843684053=39, srv1691460081=34, srv730230343=90, srv1185392749=10, srv39830031=69, srv1149272656=9, srv642567785=86, srv1446815809=25, srv1449379238=26, srv1946947354=46, srv311338758=63, srv1252358455=14, srv185470342=42, srv1210003051=13, srv1134444174=7, srv534436617=74, srv1041039980=0, srv1120332450=5, srv581928158=78, srv592738608=81, srv1410438513=23, srv1346733897=21, srv911253542=98, srv596589458=83, srv2139247535=55, srv725845781=89, srv1268353618=15, srv1421929733=24, srv330380609=65, srv586355411=79, srv246274323=57, srv3087209=62, srv1911678238=45, srv896279324=97, srv1858758779=43, srv1625798352=30, srv1147052269=8, srv2093115318=52, srv626834530=85, srv28995972=60, srv2074378551=50, srv1311143374=17, srv1705413790=36, srv280632188=58, srv1668025679=32, srv354199743=67, srv349477235=66, srv1877797389=44, srv916784157=99, srv548782692=76, srv1351541090=22, srv880097834=95, srv720834968=88, srv418157925=70, srv2104926888=53, srv1093099248=1, srv519655418=73, srv1103506571=2, srv2085498089=51, srv679219198=87, srv835838702=94, srv1757607884=37, srv1132763457=6, srv1119190866=4, srv1313651546=18, srv1846067276=40, srv1187077157=11, srv556862115=77, srv168000313=33, srv757810445=91, srv16510149=31, srv1316328634=19, srv280845145=59, srv1704129133=35, srv1342143217=20, srv1461508801=27, srv2063704772=49, srv548635289=75, srv295724291=61, srv2114148340=54, srv829302628=93, srv586488169=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:14,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:14,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:14,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:14,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:14,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:14,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:14,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:14,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-23 11:53:14,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-23 11:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-23 11:53:14,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-23 11:53:14,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-23 11:53:14,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-23 11:53:14,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-23 11:53:14,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-23 11:53:14,779 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05429271787602717, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:14,780 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-23 11:53:14,780 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18025570019798592, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05429271787602717, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-23 11:53:15,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations.  Found a solution that moves 2639 regions; Going from a computed imbalance of 0.18025570019798592 to a new imbalance of 0.014275625370004675. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018254686255058102); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02941631971066172, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2932222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018254686255058102); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:15,169 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1605379565:76 , srv452255616:77 , srv1041039980:79 , srv725845781:79 , srv1911678238:81 , srv295724291:81 , srv311338758:81 , srv812286763:81 , srv1187077157:82 , srv1625798352:82 , srv1252358455:83 , srv1311143374:83 , srv1316328634:83 , srv1351541090:83 , srv280845145:83 , srv548782692:83 , srv896279324:83 , srv1119190866:84 , srv1149272656:84 , srv1276124647:84 , srv1342143217:84 , srv1421929733:84 , srv1446815809:84 , srv1602985424:84 , srv1843684053:84 , srv280632188:84 , srv330380609:84 , srv354199743:84 , srv39830031:84 , srv594415309:84 , srv596589458:84 , srv626834530:84 , srv829302628:84 , srv1093099248:85 , srv1210003051:85 , srv1346733897:85 , srv1461508801:85 , srv1668025679:85 , srv1846067276:85 , srv2085498089:85 , srv2104926888:85 , srv2114148340:85 , srv556862115:85 , srv835838702:85 , srv880097834:85 , srv894489445:85 , srv916784157:85 , srv1132763457:86 , srv1449379238:86 , srv168000313:86 , srv1852114209:86 , srv1990996863:86 , srv2063704772:86 , srv2093115318:86 , srv246274323:86 , srv349477235:86 , srv418157925:86 , srv519655418:86 , srv642567785:86 , srv720834968:86 , srv1103506571:87 , srv1120332450:87 , srv1147052269:87 , srv1209647324:87 , srv1820266700:87 , srv185470342:87 , srv1858758779:87 , srv1946947354:87 , srv2028222241:87 , srv386706315:87 , srv586355411:87 , srv622975529:87 , srv1313651546:88 , srv1410438513:88 , srv16510149:88 , srv1705413790:88 , srv1877797389:88 , srv2139247535:88 , srv3087209:88 , srv592738608:88 , srv1118398625:89 , srv1134444174:89 , srv1185392749:89 , srv1691460081:89 , srv227322633:89 , srv327164324:89 , srv548635289:89 , srv586488169:89 , srv757810445:89 , srv911253542:89 , srv2074378551:90 , srv28995972:90 , srv471799143:90 , srv730230343:90 , srv1704129133:91 , srv534436617:91 , srv1268353618:92 , srv581928158:92 , srv679219198:92 , srv1757607884:512 }
2023-09-23 11:53:15,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:15,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852114209=41, srv594415309=82, srv327164324=64, srv1605379565=29, srv227322633=56, srv812286763=92, srv1820266700=38, srv452255616=71, srv894489445=96, srv1209647324=12, srv1602985424=28, srv386706315=68, srv471799143=72, srv1118398625=3, srv622975529=84, srv1276124647=16, srv2028222241=48, srv1990996863=47, srv1843684053=39, srv1691460081=34, srv730230343=90, srv1185392749=10, srv39830031=69, srv1149272656=9, srv642567785=86, srv1446815809=25, srv1449379238=26, srv1946947354=46, srv311338758=63, srv1252358455=14, srv185470342=42, srv1210003051=13, srv1134444174=7, srv534436617=74, srv1041039980=0, srv1120332450=5, srv581928158=78, srv592738608=81, srv1410438513=23, srv1346733897=21, srv911253542=98, srv596589458=83, srv2139247535=55, srv725845781=89, srv1268353618=15, srv1421929733=24, srv330380609=65, srv586355411=79, srv246274323=57, srv3087209=62, srv1911678238=45, srv896279324=97, srv1858758779=43, srv1625798352=30, srv1147052269=8, srv2093115318=52, srv626834530=85, srv28995972=60, srv2074378551=50, srv1311143374=17, srv1705413790=36, srv280632188=58, srv1668025679=32, srv354199743=67, srv349477235=66, srv1877797389=44, srv916784157=99, srv548782692=76, srv1351541090=22, srv880097834=95, srv720834968=88, srv418157925=70, srv2104926888=53, srv1093099248=1, srv519655418=73, srv1103506571=2, srv2085498089=51, srv679219198=87, srv835838702=94, srv1757607884=37, srv1132763457=6, srv1119190866=4, srv1313651546=18, srv1846067276=40, srv1187077157=11, srv556862115=77, srv168000313=33, srv757810445=91, srv16510149=31, srv1316328634=19, srv280845145=59, srv1704129133=35, srv1342143217=20, srv1461508801=27, srv2063704772=49, srv548635289=75, srv295724291=61, srv2114148340=54, srv829302628=93, srv586488169=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:15,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:15,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:15,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-23 11:53:15,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-23 11:53:15,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-23 11:53:15,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-23 11:53:15,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-23 11:53:15,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-23 11:53:15,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-23 11:53:15,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-23 11:53:15,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-23 11:53:15,356 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018254686255058102); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02941631971066172, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018254686255058102); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:15,356 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-23 11:53:15,356 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01425714669705208, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018254686255058102); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02941631971066172, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018254686255058102); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-23 11:53:15,809 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 583 ms to try 5000 different iterations.  Found a solution that moves 914 regions; Going from a computed imbalance of 0.01425714669705208 to a new imbalance of 4.069535746377176E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003881265576024124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.10155555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003881265576024124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:15,811 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv679219198:80 , srv1602985424:82 , srv2104926888:82 , srv581928158:82 , srv730230343:82 , srv1410438513:83 , srv1705413790:83 , srv1858758779:83 , srv2139247535:83 , srv227322633:83 , srv471799143:83 , srv519655418:83 , srv592738608:83 , srv1276124647:84 , srv1346733897:84 , srv1446815809:84 , srv1449379238:84 , srv16510149:84 , srv168000313:84 , srv1704129133:84 , srv1820266700:84 , srv2063704772:84 , srv3087209:84 , srv586355411:84 , srv586488169:84 , srv757810445:84 , srv1041039980:85 , srv1118398625:85 , srv1134444174:85 , srv1185392749:85 , srv1209647324:85 , srv1252358455:85 , srv1268353618:85 , srv1313651546:85 , srv1461508801:85 , srv1668025679:85 , srv1691460081:85 , srv1843684053:85 , srv1877797389:85 , srv1911678238:85 , srv2074378551:85 , srv280632188:85 , srv28995972:85 , srv295724291:85 , srv327164324:85 , srv330380609:85 , srv548635289:85 , srv548782692:85 , srv594415309:85 , srv725845781:85 , srv835838702:85 , srv894489445:85 , srv911253542:85 , srv1103506571:86 , srv1119190866:86 , srv1132763457:86 , srv1147052269:86 , srv1149272656:86 , srv1187077157:86 , srv1351541090:86 , srv1625798352:86 , srv1846067276:86 , srv1852114209:86 , srv1946947354:86 , srv2028222241:86 , srv2085498089:86 , srv2114148340:86 , srv246274323:86 , srv349477235:86 , srv386706315:86 , srv534436617:86 , srv622975529:86 , srv626834530:86 , srv812286763:86 , srv829302628:86 , srv896279324:86 , srv1120332450:87 , srv1210003051:87 , srv1311143374:87 , srv1316328634:87 , srv1342143217:87 , srv1421929733:87 , srv185470342:87 , srv1990996863:87 , srv280845145:87 , srv311338758:87 , srv354199743:87 , srv39830031:87 , srv418157925:87 , srv452255616:87 , srv556862115:87 , srv596589458:87 , srv642567785:87 , srv720834968:87 , srv1093099248:88 , srv1605379565:88 , srv2093115318:88 , srv880097834:88 , srv916784157:89 , srv1757607884:556 }
2023-09-23 11:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852114209=41, srv594415309=82, srv327164324=64, srv1605379565=29, srv227322633=56, srv812286763=92, srv1820266700=38, srv452255616=71, srv894489445=96, srv1209647324=12, srv1602985424=28, srv386706315=68, srv471799143=72, srv1118398625=3, srv622975529=84, srv1276124647=16, srv2028222241=48, srv1990996863=47, srv1843684053=39, srv1691460081=34, srv730230343=90, srv1185392749=10, srv39830031=69, srv1149272656=9, srv642567785=86, srv1446815809=25, srv1449379238=26, srv1946947354=46, srv311338758=63, srv1252358455=14, srv185470342=42, srv1210003051=13, srv1134444174=7, srv534436617=74, srv1041039980=0, srv1120332450=5, srv581928158=78, srv592738608=81, srv1410438513=23, srv1346733897=21, srv911253542=98, srv596589458=83, srv2139247535=55, srv725845781=89, srv1268353618=15, srv1421929733=24, srv330380609=65, srv586355411=79, srv246274323=57, srv3087209=62, srv1911678238=45, srv896279324=97, srv1858758779=43, srv1625798352=30, srv1147052269=8, srv2093115318=52, srv626834530=85, srv28995972=60, srv2074378551=50, srv1311143374=17, srv1705413790=36, srv280632188=58, srv1668025679=32, srv354199743=67, srv349477235=66, srv1877797389=44, srv916784157=99, srv548782692=76, srv1351541090=22, srv880097834=95, srv720834968=88, srv418157925=70, srv2104926888=53, srv1093099248=1, srv519655418=73, srv1103506571=2, srv2085498089=51, srv679219198=87, srv835838702=94, srv1757607884=37, srv1132763457=6, srv1119190866=4, srv1313651546=18, srv1846067276=40, srv1187077157=11, srv556862115=77, srv168000313=33, srv757810445=91, srv16510149=31, srv1316328634=19, srv280845145=59, srv1704129133=35, srv1342143217=20, srv1461508801=27, srv2063704772=49, srv548635289=75, srv295724291=61, srv2114148340=54, srv829302628=93, srv586488169=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-23 11:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-23 11:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-23 11:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-23 11:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-23 11:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-23 11:53:15,966 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003881265576024124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003881265576024124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:15,966 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-23 11:53:15,967 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.0055360984852736E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003881265576024124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003881265576024124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-23 11:53:16,281 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 428 ms to try 5000 different iterations.  Found a solution that moves 183 regions; Going from a computed imbalance of 4.0055360984852736E-4 to a new imbalance of 1.0190449703488954E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.02033333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:16,282 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1210003051:83 , srv1268353618:83 , srv1990996863:83 , srv2063704772:83 , srv354199743:83 , srv418157925:83 , srv642567785:83 , srv916784157:83 , srv1041039980:84 , srv1093099248:84 , srv1103506571:84 , srv1119190866:84 , srv1120332450:84 , srv1132763457:84 , srv1252358455:84 , srv1313651546:84 , srv1316328634:84 , srv1351541090:84 , srv1410438513:84 , srv1605379565:84 , srv1625798352:84 , srv16510149:84 , srv168000313:84 , srv1691460081:84 , srv1704129133:84 , srv1820266700:84 , srv1846067276:84 , srv1852114209:84 , srv185470342:84 , srv1858758779:84 , srv1877797389:84 , srv1911678238:84 , srv1946947354:84 , srv2074378551:84 , srv2085498089:84 , srv2093115318:84 , srv280632188:84 , srv28995972:84 , srv311338758:84 , srv349477235:84 , srv386706315:84 , srv39830031:84 , srv452255616:84 , srv471799143:84 , srv548635289:84 , srv556862115:84 , srv586488169:84 , srv592738608:84 , srv596589458:84 , srv622975529:84 , srv679219198:84 , srv720834968:84 , srv725845781:84 , srv829302628:84 , srv835838702:84 , srv880097834:84 , srv896279324:84 , srv911253542:84 , srv1118398625:85 , srv1134444174:85 , srv1147052269:85 , srv1149272656:85 , srv1185392749:85 , srv1187077157:85 , srv1209647324:85 , srv1276124647:85 , srv1311143374:85 , srv1342143217:85 , srv1346733897:85 , srv1421929733:85 , srv1446815809:85 , srv1449379238:85 , srv1461508801:85 , srv1602985424:85 , srv1668025679:85 , srv1705413790:85 , srv1843684053:85 , srv2028222241:85 , srv2114148340:85 , srv2139247535:85 , srv227322633:85 , srv246274323:85 , srv280845145:85 , srv295724291:85 , srv3087209:85 , srv327164324:85 , srv330380609:85 , srv519655418:85 , srv534436617:85 , srv548782692:85 , srv581928158:85 , srv586355411:85 , srv594415309:85 , srv626834530:85 , srv730230343:85 , srv757810445:85 , srv812286763:85 , srv894489445:85 , srv2104926888:86 , srv1757607884:650 }
2023-09-23 11:53:16,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:16,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852114209=41, srv594415309=82, srv327164324=64, srv1605379565=29, srv227322633=56, srv812286763=92, srv1820266700=38, srv452255616=71, srv894489445=96, srv1209647324=12, srv1602985424=28, srv386706315=68, srv471799143=72, srv1118398625=3, srv622975529=84, srv1276124647=16, srv2028222241=48, srv1990996863=47, srv1843684053=39, srv1691460081=34, srv730230343=90, srv1185392749=10, srv39830031=69, srv1149272656=9, srv642567785=86, srv1446815809=25, srv1449379238=26, srv1946947354=46, srv311338758=63, srv1252358455=14, srv185470342=42, srv1210003051=13, srv1134444174=7, srv534436617=74, srv1041039980=0, srv1120332450=5, srv581928158=78, srv592738608=81, srv1410438513=23, srv1346733897=21, srv911253542=98, srv596589458=83, srv2139247535=55, srv725845781=89, srv1268353618=15, srv1421929733=24, srv330380609=65, srv586355411=79, srv246274323=57, srv3087209=62, srv1911678238=45, srv896279324=97, srv1858758779=43, srv1625798352=30, srv1147052269=8, srv2093115318=52, srv626834530=85, srv28995972=60, srv2074378551=50, srv1311143374=17, srv1705413790=36, srv280632188=58, srv1668025679=32, srv354199743=67, srv349477235=66, srv1877797389=44, srv916784157=99, srv548782692=76, srv1351541090=22, srv880097834=95, srv720834968=88, srv418157925=70, srv2104926888=53, srv1093099248=1, srv519655418=73, srv1103506571=2, srv2085498089=51, srv679219198=87, srv835838702=94, srv1757607884=37, srv1132763457=6, srv1119190866=4, srv1313651546=18, srv1846067276=40, srv1187077157=11, srv556862115=77, srv168000313=33, srv757810445=91, srv16510149=31, srv1316328634=19, srv280845145=59, srv1704129133=35, srv1342143217=20, srv1461508801=27, srv2063704772=49, srv548635289=75, srv295724291=61, srv2114148340=54, srv829302628=93, srv586488169=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:16,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:16,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:16,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:16,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-23 11:53:16,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-23 11:53:16,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-23 11:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-23 11:53:16,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-23 11:53:16,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-23 11:53:16,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-23 11:53:16,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-23 11:53:16,418 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.909056315718909E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:16,418 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1210003051:83 , srv1268353618:83 , srv1990996863:83 , srv2063704772:83 , srv354199743:83 , srv418157925:83 , srv642567785:83 , srv916784157:83 , srv1041039980:84 , srv1093099248:84 , srv1103506571:84 , srv1119190866:84 , srv1120332450:84 , srv1132763457:84 , srv1252358455:84 , srv1313651546:84 , srv1316328634:84 , srv1351541090:84 , srv1410438513:84 , srv1605379565:84 , srv1625798352:84 , srv16510149:84 , srv168000313:84 , srv1691460081:84 , srv1704129133:84 , srv1820266700:84 , srv1846067276:84 , srv1852114209:84 , srv185470342:84 , srv1858758779:84 , srv1877797389:84 , srv1911678238:84 , srv1946947354:84 , srv2074378551:84 , srv2085498089:84 , srv2093115318:84 , srv280632188:84 , srv28995972:84 , srv311338758:84 , srv349477235:84 , srv386706315:84 , srv39830031:84 , srv452255616:84 , srv471799143:84 , srv548635289:84 , srv556862115:84 , srv586488169:84 , srv592738608:84 , srv596589458:84 , srv622975529:84 , srv679219198:84 , srv720834968:84 , srv725845781:84 , srv829302628:84 , srv835838702:84 , srv880097834:84 , srv896279324:84 , srv911253542:84 , srv1118398625:85 , srv1134444174:85 , srv1147052269:85 , srv1149272656:85 , srv1185392749:85 , srv1187077157:85 , srv1209647324:85 , srv1276124647:85 , srv1311143374:85 , srv1342143217:85 , srv1346733897:85 , srv1421929733:85 , srv1446815809:85 , srv1449379238:85 , srv1461508801:85 , srv1602985424:85 , srv1668025679:85 , srv1705413790:85 , srv1843684053:85 , srv2028222241:85 , srv2114148340:85 , srv2139247535:85 , srv227322633:85 , srv246274323:85 , srv280845145:85 , srv295724291:85 , srv3087209:85 , srv327164324:85 , srv330380609:85 , srv519655418:85 , srv534436617:85 , srv548782692:85 , srv581928158:85 , srv586355411:85 , srv594415309:85 , srv626834530:85 , srv730230343:85 , srv757810445:85 , srv812286763:85 , srv894489445:85 , srv2104926888:86 , srv1757607884:650 }
2023-09-23 11:53:16,453 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=319 (was 259) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=2526 (was 3117)
2023-09-23 11:53:16,463 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=319, ProcessCount=172, AvailableMemoryMB=2526
2023-09-23 11:53:16,464 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-23 11:53:16,464 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-23 11:53:16,472 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1148299828:84 , srv1162174427:84 , srv1317554323:84 , srv1342893653:84 , srv1373303128:84 , srv1419051985:84 , srv1685213888:84 , srv1686613250:84 , srv1701265890:84 , srv1702398635:84 , srv1720904262:84 , srv1765894339:84 , srv1871410889:84 , srv1872166568:84 , srv1883339527:84 , srv1903153734:84 , srv245810264:84 , srv263966870:84 , srv399746684:84 , srv43460557:84 , srv469670581:84 , srv53853169:84 , srv592824474:84 , srv63141054:84 , srv635297003:84 , srv650374986:84 , srv672924085:84 , srv816179051:84 , srv844863168:84 , srv2020697678:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-23 11:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263966870=18, srv1162174427=1, srv1317554323=2, srv245810264=17, srv1342893653=3, srv399746684=19, srv1701265890=8, srv1685213888=6, srv1765894339=11, srv1871410889=12, srv2020697678=16, srv1373303128=4, srv592824474=23, srv1883339527=14, srv1720904262=10, srv635297003=25, srv844863168=29, srv1903153734=15, srv1686613250=7, srv1872166568=13, srv672924085=27, srv1419051985=5, srv1148299828=0, srv469670581=21, srv53853169=22, srv1702398635=9, srv43460557=20, srv63141054=24, srv650374986=26, srv816179051=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-23 11:53:16,497 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.087718663581487, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:53:16,498 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912469441908948, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.087718663581487, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-23 11:55:16,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5736553 different iterations.  Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912469441908948 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:55:16,491 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1148299828:90 , srv1162174427:90 , srv1317554323:90 , srv1342893653:90 , srv1373303128:90 , srv1419051985:90 , srv1685213888:90 , srv1686613250:90 , srv1701265890:90 , srv1702398635:90 , srv1720904262:90 , srv1765894339:90 , srv1871410889:90 , srv1872166568:90 , srv1883339527:90 , srv1903153734:90 , srv2020697678:90 , srv245810264:90 , srv263966870:90 , srv399746684:90 , srv43460557:90 , srv469670581:90 , srv53853169:90 , srv592824474:90 , srv63141054:90 , srv635297003:90 , srv650374986:90 , srv672924085:90 , srv816179051:90 , srv844863168:90 }
2023-09-23 11:55:16,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-23 11:55:16,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263966870=18, srv1162174427=1, srv1317554323=2, srv245810264=17, srv1342893653=3, srv399746684=19, srv1701265890=8, srv1685213888=6, srv1765894339=11, srv1871410889=12, srv2020697678=16, srv1373303128=4, srv592824474=23, srv1883339527=14, srv1720904262=10, srv635297003=25, srv844863168=29, srv1903153734=15, srv1686613250=7, srv1872166568=13, srv672924085=27, srv1419051985=5, srv1148299828=0, srv469670581=21, srv53853169=22, srv1702398635=9, srv43460557=20, srv63141054=24, srv650374986=26, srv816179051=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-23 11:55:16,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-23 11:55:16,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-23 11:55:16,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-23 11:55:16,509 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-23 11:55:16,509 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1148299828:90 , srv1162174427:90 , srv1317554323:90 , srv1342893653:90 , srv1373303128:90 , srv1419051985:90 , srv1685213888:90 , srv1686613250:90 , srv1701265890:90 , srv1702398635:90 , srv1720904262:90 , srv1765894339:90 , srv1871410889:90 , srv1872166568:90 , srv1883339527:90 , srv1903153734:90 , srv2020697678:90 , srv245810264:90 , srv263966870:90 , srv399746684:90 , srv43460557:90 , srv469670581:90 , srv53853169:90 , srv592824474:90 , srv63141054:90 , srv635297003:90 , srv650374986:90 , srv672924085:90 , srv816179051:90 , srv844863168:90 }
2023-09-23 11:55:16,521 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=142 (was 319), ProcessCount=171 (was 172), AvailableMemoryMB=1738 (was 2526)