2023-11-17 19:52:49,393 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-17 19:52:49,704 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-17 19:52:49,810 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-17 19:52:49,876 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-17 19:52:49,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-17 19:52:49,895 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=175, AvailableMemoryMB=4275
2023-11-17 19:52:49,898 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-17 19:52:49,898 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-17 19:52:50,003 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022464852:84 , srv1030214145:84 , srv1030697889:84 , srv103328837:84 , srv1053646610:84 , srv1064330253:84 , srv1073378444:84 , srv1094081771:84 , srv111309903:84 , srv1113533393:84 , srv1137393496:84 , srv1171812867:84 , srv1175094958:84 , srv1207214682:84 , srv120746057:84 , srv1235034004:84 , srv1242438855:84 , srv1244935290:84 , srv1271206825:84 , srv1303346323:84 , srv1326023051:84 , srv1332119282:84 , srv1334175522:84 , srv1335914510:84 , srv1346441738:84 , srv1355215298:84 , srv1367253421:84 , srv1372726230:84 , srv1389320164:84 , srv1403326166:84 , srv1469438706:84 , srv1473818121:84 , srv1481968388:84 , srv1499649765:84 , srv1509618917:84 , srv1540670536:84 , srv1568479659:84 , srv1586447321:84 , srv1608814280:84 , srv1616484850:84 , srv1620915649:84 , srv162210138:84 , srv1644935070:84 , srv1650437095:84 , srv1714937137:84 , srv1716535937:84 , srv1722430794:84 , srv1787454980:84 , srv1795441712:84 , srv1809664420:84 , srv1817138069:84 , srv1834796784:84 , srv1850707579:84 , srv188147830:84 , srv1894417345:84 , srv1904051029:84 , srv1914007702:84 , srv197060318:84 , srv2006159303:84 , srv2020323634:84 , srv20578720:84 , srv2100639635:84 , srv210129325:84 , srv2124495009:84 , srv2125051468:84 , srv2141360814:84 , srv216943000:84 , srv226561549:84 , srv239485881:84 , srv282149162:84 , srv330543973:84 , srv359050944:84 , srv499838464:84 , srv54115014:84 , srv552209071:84 , srv573234305:84 , srv585798164:84 , srv608143639:84 , srv60837499:84 , srv612891368:84 , srv645356477:84 , srv663770124:84 , srv674335482:84 , srv715103551:84 , srv72585001:84 , srv726835124:84 , srv731375628:84 , srv732308977:84 , srv777562373:84 , srv779809382:84 , srv787466352:84 , srv791659023:84 , srv820928841:84 , srv840434691:84 , srv855554626:84 , srv862572127:84 , srv879473460:84 , srv89972053:84 , srv923042643:84 , srv1424000280:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-17 19:52:50,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:50,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv645356477=81, srv197060318=58, srv1326023051=20, srv1620915649=41, srv1914007702=57, srv1616484850=40, srv330543973=71, srv1030214145=1, srv1235034004=15, srv1904051029=56, srv585798164=77, srv1335914510=23, srv923042643=99, srv1608814280=39, srv1244935290=17, srv1367253421=26, srv1271206825=18, srv1644935070=43, srv1650437095=44, srv779809382=90, srv1424000280=30, srv715103551=84, srv1714937137=45, srv499838464=73, srv1332119282=21, srv552209071=75, srv1022464852=0, srv1809664420=50, srv862572127=96, srv1207214682=13, srv1137393496=10, srv2125051468=65, srv1722430794=47, srv2006159303=59, srv820928841=93, srv1894417345=55, srv1355215298=25, srv731375628=87, srv1499649765=34, srv1716535937=46, srv89972053=98, srv1586447321=38, srv1473818121=32, srv573234305=76, srv1064330253=5, srv1795441712=49, srv359050944=72, srv72585001=85, srv20578720=61, srv1481968388=33, srv1334175522=22, srv188147830=54, srv732308977=88, srv1568479659=37, srv674335482=83, srv777562373=89, srv1030697889=2, srv1372726230=27, srv60837499=79, srv1834796784=52, srv2020323634=60, srv239485881=69, srv1540670536=36, srv216943000=67, srv726835124=86, srv1389320164=28, srv162210138=42, srv855554626=95, srv879473460=97, srv1850707579=53, srv1303346323=19, srv791659023=92, srv1817138069=51, srv1175094958=12, srv120746057=14, srv1469438706=31, srv282149162=70, srv2124495009=64, srv612891368=80, srv103328837=3, srv54115014=74, srv111309903=8, srv840434691=94, srv2100639635=62, srv663770124=82, srv1073378444=6, srv226561549=68, srv787466352=91, srv608143639=78, srv1787454980=48, srv1113533393=9, srv1053646610=4, srv1509618917=35, srv1242438855=16, srv210129325=63, srv1171812867=11, srv1346441738=24, srv1094081771=7, srv1403326166=29, srv2141360814=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:50,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-17 19:52:50,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-17 19:52:50,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-17 19:52:50,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-17 19:52:50,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-17 19:52:50,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-17 19:52:50,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-17 19:52:50,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-17 19:52:50,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-17 19:52:50,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-17 19:52:50,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-17 19:52:50,263 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:50,263 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-17 19:52:50,263 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-17 19:52:50,570 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 450 ms to try 5000 different iterations.  Found a solution that moves 4709 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1786626103008565. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037298070032952976, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055046149396773394, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5232222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037298070032952976, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:50,592 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv777562373:81 , srv359050944:82 , srv552209071:82 , srv645356477:82 , srv1030697889:83 , srv1175094958:83 , srv1355215298:83 , srv1389320164:83 , srv1568479659:83 , srv1608814280:83 , srv1714937137:83 , srv1722430794:83 , srv1914007702:83 , srv2124495009:83 , srv791659023:83 , srv103328837:84 , srv1235034004:84 , srv1332119282:84 , srv1346441738:84 , srv1403326166:84 , srv1509618917:84 , srv1795441712:84 , srv226561549:84 , srv499838464:84 , srv573234305:84 , srv60837499:84 , srv1030214145:85 , srv1171812867:85 , srv1242438855:85 , srv1326023051:85 , srv1473818121:85 , srv1540670536:85 , srv1620915649:85 , srv1644935070:85 , srv210129325:85 , srv2141360814:85 , srv731375628:85 , srv732308977:85 , srv855554626:85 , srv879473460:85 , srv1022464852:86 , srv1053646610:86 , srv1073378444:86 , srv1094081771:86 , srv1137393496:86 , srv120746057:86 , srv1334175522:86 , srv1367253421:86 , srv1481968388:86 , srv1809664420:86 , srv1834796784:86 , srv2100639635:86 , srv330543973:86 , srv612891368:86 , srv1113533393:87 , srv1787454980:87 , srv1850707579:87 , srv197060318:87 , srv2020323634:87 , srv282149162:87 , srv608143639:87 , srv715103551:87 , srv923042643:87 , srv111309903:88 , srv1303346323:88 , srv162210138:88 , srv1650437095:88 , srv1817138069:88 , srv585798164:88 , srv779809382:88 , srv787466352:88 , srv862572127:88 , srv1207214682:89 , srv1335914510:89 , srv1372726230:89 , srv1586447321:89 , srv1616484850:89 , srv1716535937:89 , srv1894417345:89 , srv216943000:89 , srv674335482:89 , srv89972053:89 , srv1064330253:90 , srv1244935290:90 , srv188147830:90 , srv2125051468:90 , srv239485881:90 , srv840434691:90 , srv1499649765:91 , srv2006159303:91 , srv20578720:91 , srv663770124:91 , srv726835124:91 , srv1469438706:92 , srv1271206825:93 , srv1904051029:93 , srv72585001:93 , srv54115014:95 , srv820928841:95 , srv1424000280:421 }
2023-11-17 19:52:50,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:50,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv645356477=81, srv197060318=58, srv1326023051=20, srv1620915649=41, srv1914007702=57, srv1616484850=40, srv330543973=71, srv1030214145=1, srv1235034004=15, srv1904051029=56, srv585798164=77, srv1335914510=23, srv923042643=99, srv1608814280=39, srv1244935290=17, srv1367253421=26, srv1271206825=18, srv1644935070=43, srv1650437095=44, srv779809382=90, srv1424000280=30, srv715103551=84, srv1714937137=45, srv499838464=73, srv1332119282=21, srv552209071=75, srv1022464852=0, srv1809664420=50, srv862572127=96, srv1207214682=13, srv1137393496=10, srv2125051468=65, srv1722430794=47, srv2006159303=59, srv820928841=93, srv1894417345=55, srv1355215298=25, srv731375628=87, srv1499649765=34, srv1716535937=46, srv89972053=98, srv1586447321=38, srv1473818121=32, srv573234305=76, srv1064330253=5, srv1795441712=49, srv359050944=72, srv72585001=85, srv20578720=61, srv1481968388=33, srv1334175522=22, srv188147830=54, srv732308977=88, srv1568479659=37, srv674335482=83, srv777562373=89, srv1030697889=2, srv1372726230=27, srv60837499=79, srv1834796784=52, srv2020323634=60, srv239485881=69, srv1540670536=36, srv216943000=67, srv726835124=86, srv1389320164=28, srv162210138=42, srv855554626=95, srv879473460=97, srv1850707579=53, srv1303346323=19, srv791659023=92, srv1817138069=51, srv1175094958=12, srv120746057=14, srv1469438706=31, srv282149162=70, srv2124495009=64, srv612891368=80, srv103328837=3, srv54115014=74, srv111309903=8, srv840434691=94, srv2100639635=62, srv663770124=82, srv1073378444=6, srv226561549=68, srv787466352=91, srv608143639=78, srv1787454980=48, srv1113533393=9, srv1053646610=4, srv1509618917=35, srv1242438855=16, srv210129325=63, srv1171812867=11, srv1346441738=24, srv1094081771=7, srv1403326166=29, srv2141360814=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:50,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:50,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:50,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-17 19:52:50,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-17 19:52:50,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-17 19:52:50,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-17 19:52:50,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-17 19:52:50,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-17 19:52:50,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-17 19:52:50,739 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037298070032952976, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055046149396773394, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037298070032952976, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:50,739 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-17 19:52:50,739 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17862963717810784, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037298070032952976, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055046149396773394, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037298070032952976, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-17 19:52:50,930 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations.  Found a solution that moves 2596 regions; Going from a computed imbalance of 0.17862963717810784 to a new imbalance of 0.01446975612647156. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0209358309146495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03474130788593228, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28844444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0209358309146495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:50,941 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv54115014:76 , srv72585001:77 , srv820928841:78 , srv1469438706:79 , srv1904051029:79 , srv1499649765:80 , srv188147830:80 , srv1271206825:81 , srv1586447321:81 , srv20578720:81 , srv840434691:81 , srv89972053:81 , srv1244935290:82 , srv1650437095:82 , srv585798164:82 , srv663770124:82 , srv111309903:83 , srv1207214682:83 , srv1335914510:83 , srv1787454980:83 , srv2006159303:83 , srv2020323634:83 , srv2125051468:83 , srv239485881:83 , srv674335482:83 , srv715103551:83 , srv779809382:83 , srv1372726230:84 , srv1716535937:84 , srv1817138069:84 , srv1894417345:84 , srv282149162:84 , srv787466352:84 , srv862572127:84 , srv1064330253:85 , srv1073378444:85 , srv1113533393:85 , srv1137393496:85 , srv120746057:85 , srv1616484850:85 , srv1850707579:85 , srv216943000:85 , srv608143639:85 , srv612891368:85 , srv726835124:85 , srv731375628:85 , srv1022464852:86 , srv1094081771:86 , srv1303346323:86 , srv1334175522:86 , srv1473818121:86 , srv1509618917:86 , srv197060318:86 , srv2100639635:86 , srv330543973:86 , srv499838464:86 , srv879473460:86 , srv1171812867:87 , srv1235034004:87 , srv1326023051:87 , srv1389320164:87 , srv1481968388:87 , srv1568479659:87 , srv1795441712:87 , srv1834796784:87 , srv2141360814:87 , srv226561549:87 , srv732308977:87 , srv791659023:87 , srv855554626:87 , srv1030214145:88 , srv1053646610:88 , srv1242438855:88 , srv1332119282:88 , srv1346441738:88 , srv1367253421:88 , srv1540670536:88 , srv1620915649:88 , srv162210138:88 , srv1644935070:88 , srv1809664420:88 , srv210129325:88 , srv2124495009:88 , srv573234305:88 , srv103328837:89 , srv1175094958:89 , srv1608814280:89 , srv1714937137:89 , srv552209071:89 , srv923042643:89 , srv1030697889:90 , srv1355215298:90 , srv1403326166:90 , srv1722430794:90 , srv60837499:90 , srv645356477:90 , srv1914007702:91 , srv359050944:92 , srv777562373:92 , srv1424000280:539 }
2023-11-17 19:52:50,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:50,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv645356477=81, srv197060318=58, srv1326023051=20, srv1620915649=41, srv1914007702=57, srv1616484850=40, srv330543973=71, srv1030214145=1, srv1235034004=15, srv1904051029=56, srv585798164=77, srv1335914510=23, srv923042643=99, srv1608814280=39, srv1244935290=17, srv1367253421=26, srv1271206825=18, srv1644935070=43, srv1650437095=44, srv779809382=90, srv1424000280=30, srv715103551=84, srv1714937137=45, srv499838464=73, srv1332119282=21, srv552209071=75, srv1022464852=0, srv1809664420=50, srv862572127=96, srv1207214682=13, srv1137393496=10, srv2125051468=65, srv1722430794=47, srv2006159303=59, srv820928841=93, srv1894417345=55, srv1355215298=25, srv731375628=87, srv1499649765=34, srv1716535937=46, srv89972053=98, srv1586447321=38, srv1473818121=32, srv573234305=76, srv1064330253=5, srv1795441712=49, srv359050944=72, srv72585001=85, srv20578720=61, srv1481968388=33, srv1334175522=22, srv188147830=54, srv732308977=88, srv1568479659=37, srv674335482=83, srv777562373=89, srv1030697889=2, srv1372726230=27, srv60837499=79, srv1834796784=52, srv2020323634=60, srv239485881=69, srv1540670536=36, srv216943000=67, srv726835124=86, srv1389320164=28, srv162210138=42, srv855554626=95, srv879473460=97, srv1850707579=53, srv1303346323=19, srv791659023=92, srv1817138069=51, srv1175094958=12, srv120746057=14, srv1469438706=31, srv282149162=70, srv2124495009=64, srv612891368=80, srv103328837=3, srv54115014=74, srv111309903=8, srv840434691=94, srv2100639635=62, srv663770124=82, srv1073378444=6, srv226561549=68, srv787466352=91, srv608143639=78, srv1787454980=48, srv1113533393=9, srv1053646610=4, srv1509618917=35, srv1242438855=16, srv210129325=63, srv1171812867=11, srv1346441738=24, srv1094081771=7, srv1403326166=29, srv2141360814=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:50,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:50,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:50,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:50,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:50,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:50,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:50,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-17 19:52:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-17 19:52:50,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-17 19:52:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-17 19:52:50,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-17 19:52:50,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-17 19:52:50,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-17 19:52:50,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-17 19:52:51,054 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0209358309146495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03474130788593228, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0209358309146495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,054 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-17 19:52:51,054 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014451578545954339, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0209358309146495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03474130788593228, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0209358309146495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-17 19:52:51,269 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations.  Found a solution that moves 1020 regions; Going from a computed imbalance of 0.014451578545954339 to a new imbalance of 4.326263317300129E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004436018949768876); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.11333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004436018949768876); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,271 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv162210138:82 , srv726835124:82 , srv923042643:82 , srv1064330253:83 , srv1303346323:83 , srv1367253421:83 , srv1403326166:83 , srv1809664420:83 , srv60837499:83 , srv1030697889:84 , srv1053646610:84 , srv1271206825:84 , srv1372726230:84 , srv1616484850:84 , srv1644935070:84 , srv1722430794:84 , srv1914007702:84 , srv210129325:84 , srv216943000:84 , srv359050944:84 , srv608143639:84 , srv663770124:84 , srv820928841:84 , srv1022464852:85 , srv1030214145:85 , srv103328837:85 , srv1113533393:85 , srv1207214682:85 , srv1242438855:85 , srv1244935290:85 , srv1326023051:85 , srv1355215298:85 , srv1481968388:85 , srv1540670536:85 , srv1620915649:85 , srv1716535937:85 , srv1834796784:85 , srv1894417345:85 , srv197060318:85 , srv20578720:85 , srv2100639635:85 , srv2125051468:85 , srv239485881:85 , srv330543973:85 , srv674335482:85 , srv777562373:85 , srv855554626:85 , srv1073378444:86 , srv1094081771:86 , srv1137393496:86 , srv1171812867:86 , srv1175094958:86 , srv120746057:86 , srv1235034004:86 , srv1332119282:86 , srv1334175522:86 , srv1335914510:86 , srv1346441738:86 , srv1469438706:86 , srv1608814280:86 , srv1714937137:86 , srv1817138069:86 , srv1850707579:86 , srv1904051029:86 , srv2124495009:86 , srv54115014:86 , srv552209071:86 , srv573234305:86 , srv612891368:86 , srv645356477:86 , srv732308977:86 , srv779809382:86 , srv787466352:86 , srv862572127:86 , srv879473460:86 , srv1389320164:87 , srv1473818121:87 , srv1499649765:87 , srv1509618917:87 , srv1586447321:87 , srv1795441712:87 , srv2006159303:87 , srv2141360814:87 , srv226561549:87 , srv282149162:87 , srv499838464:87 , srv585798164:87 , srv715103551:87 , srv72585001:87 , srv791659023:87 , srv840434691:87 , srv89972053:87 , srv111309903:88 , srv1568479659:88 , srv1650437095:88 , srv1787454980:88 , srv188147830:88 , srv2020323634:88 , srv731375628:88 , srv1424000280:537 }
2023-11-17 19:52:51,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:51,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv645356477=81, srv197060318=58, srv1326023051=20, srv1620915649=41, srv1914007702=57, srv1616484850=40, srv330543973=71, srv1030214145=1, srv1235034004=15, srv1904051029=56, srv585798164=77, srv1335914510=23, srv923042643=99, srv1608814280=39, srv1244935290=17, srv1367253421=26, srv1271206825=18, srv1644935070=43, srv1650437095=44, srv779809382=90, srv1424000280=30, srv715103551=84, srv1714937137=45, srv499838464=73, srv1332119282=21, srv552209071=75, srv1022464852=0, srv1809664420=50, srv862572127=96, srv1207214682=13, srv1137393496=10, srv2125051468=65, srv1722430794=47, srv2006159303=59, srv820928841=93, srv1894417345=55, srv1355215298=25, srv731375628=87, srv1499649765=34, srv1716535937=46, srv89972053=98, srv1586447321=38, srv1473818121=32, srv573234305=76, srv1064330253=5, srv1795441712=49, srv359050944=72, srv72585001=85, srv20578720=61, srv1481968388=33, srv1334175522=22, srv188147830=54, srv732308977=88, srv1568479659=37, srv674335482=83, srv777562373=89, srv1030697889=2, srv1372726230=27, srv60837499=79, srv1834796784=52, srv2020323634=60, srv239485881=69, srv1540670536=36, srv216943000=67, srv726835124=86, srv1389320164=28, srv162210138=42, srv855554626=95, srv879473460=97, srv1850707579=53, srv1303346323=19, srv791659023=92, srv1817138069=51, srv1175094958=12, srv120746057=14, srv1469438706=31, srv282149162=70, srv2124495009=64, srv612891368=80, srv103328837=3, srv54115014=74, srv111309903=8, srv840434691=94, srv2100639635=62, srv663770124=82, srv1073378444=6, srv226561549=68, srv787466352=91, srv608143639=78, srv1787454980=48, srv1113533393=9, srv1053646610=4, srv1509618917=35, srv1242438855=16, srv210129325=63, srv1171812867=11, srv1346441738=24, srv1094081771=7, srv1403326166=29, srv2141360814=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:51,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:51,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-17 19:52:51,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-17 19:52:51,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-17 19:52:51,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-17 19:52:51,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-17 19:52:51,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-17 19:52:51,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-17 19:52:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-17 19:52:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-17 19:52:51,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-17 19:52:51,396 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004436018949768876); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004436018949768876); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,396 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-17 19:52:51,397 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.254841390768684E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004436018949768876); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004436018949768876); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-17 19:52:51,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 272 ms to try 5000 different iterations.  Found a solution that moves 188 regions; Going from a computed imbalance of 4.254841390768684E-4 to a new imbalance of 1.790732862613616E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.020888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,586 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2006159303:81 , srv111309903:83 , srv1714937137:83 , srv2141360814:83 , srv282149162:83 , srv840434691:83 , srv1030214145:84 , srv1053646610:84 , srv1064330253:84 , srv1094081771:84 , srv1171812867:84 , srv1175094958:84 , srv1242438855:84 , srv1271206825:84 , srv1332119282:84 , srv1334175522:84 , srv1335914510:84 , srv1346441738:84 , srv1355215298:84 , srv1367253421:84 , srv1481968388:84 , srv1499649765:84 , srv1540670536:84 , srv1568479659:84 , srv1608814280:84 , srv1616484850:84 , srv1620915649:84 , srv162210138:84 , srv1650437095:84 , srv1716535937:84 , srv1787454980:84 , srv1795441712:84 , srv1817138069:84 , srv1834796784:84 , srv1850707579:84 , srv188147830:84 , srv1894417345:84 , srv1904051029:84 , srv1914007702:84 , srv197060318:84 , srv2020323634:84 , srv216943000:84 , srv226561549:84 , srv239485881:84 , srv359050944:84 , srv573234305:84 , srv645356477:84 , srv726835124:84 , srv731375628:84 , srv732308977:84 , srv777562373:84 , srv787466352:84 , srv862572127:84 , srv923042643:84 , srv1022464852:85 , srv1030697889:85 , srv103328837:85 , srv1073378444:85 , srv1113533393:85 , srv1137393496:85 , srv1207214682:85 , srv120746057:85 , srv1235034004:85 , srv1244935290:85 , srv1303346323:85 , srv1326023051:85 , srv1372726230:85 , srv1389320164:85 , srv1403326166:85 , srv1469438706:85 , srv1473818121:85 , srv1509618917:85 , srv1586447321:85 , srv1644935070:85 , srv1722430794:85 , srv1809664420:85 , srv20578720:85 , srv2100639635:85 , srv210129325:85 , srv2124495009:85 , srv2125051468:85 , srv330543973:85 , srv499838464:85 , srv54115014:85 , srv552209071:85 , srv585798164:85 , srv60837499:85 , srv674335482:85 , srv715103551:85 , srv72585001:85 , srv779809382:85 , srv791659023:85 , srv820928841:85 , srv855554626:85 , srv879473460:85 , srv89972053:85 , srv608143639:86 , srv612891368:86 , srv663770124:86 , srv1424000280:644 }
2023-11-17 19:52:51,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:51,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv645356477=81, srv197060318=58, srv1326023051=20, srv1620915649=41, srv1914007702=57, srv1616484850=40, srv330543973=71, srv1030214145=1, srv1235034004=15, srv1904051029=56, srv585798164=77, srv1335914510=23, srv923042643=99, srv1608814280=39, srv1244935290=17, srv1367253421=26, srv1271206825=18, srv1644935070=43, srv1650437095=44, srv779809382=90, srv1424000280=30, srv715103551=84, srv1714937137=45, srv499838464=73, srv1332119282=21, srv552209071=75, srv1022464852=0, srv1809664420=50, srv862572127=96, srv1207214682=13, srv1137393496=10, srv2125051468=65, srv1722430794=47, srv2006159303=59, srv820928841=93, srv1894417345=55, srv1355215298=25, srv731375628=87, srv1499649765=34, srv1716535937=46, srv89972053=98, srv1586447321=38, srv1473818121=32, srv573234305=76, srv1064330253=5, srv1795441712=49, srv359050944=72, srv72585001=85, srv20578720=61, srv1481968388=33, srv1334175522=22, srv188147830=54, srv732308977=88, srv1568479659=37, srv674335482=83, srv777562373=89, srv1030697889=2, srv1372726230=27, srv60837499=79, srv1834796784=52, srv2020323634=60, srv239485881=69, srv1540670536=36, srv216943000=67, srv726835124=86, srv1389320164=28, srv162210138=42, srv855554626=95, srv879473460=97, srv1850707579=53, srv1303346323=19, srv791659023=92, srv1817138069=51, srv1175094958=12, srv120746057=14, srv1469438706=31, srv282149162=70, srv2124495009=64, srv612891368=80, srv103328837=3, srv54115014=74, srv111309903=8, srv840434691=94, srv2100639635=62, srv663770124=82, srv1073378444=6, srv226561549=68, srv787466352=91, srv608143639=78, srv1787454980=48, srv1113533393=9, srv1053646610=4, srv1509618917=35, srv1242438855=16, srv210129325=63, srv1171812867=11, srv1346441738=24, srv1094081771=7, srv1403326166=29, srv2141360814=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:51,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:51,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:51,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:51,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:51,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:51,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-17 19:52:51,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-17 19:52:51,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-17 19:52:51,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-17 19:52:51,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-17 19:52:51,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-17 19:52:51,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-17 19:52:51,692 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6590924490066385E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,692 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2006159303:81 , srv111309903:83 , srv1714937137:83 , srv2141360814:83 , srv282149162:83 , srv840434691:83 , srv1030214145:84 , srv1053646610:84 , srv1064330253:84 , srv1094081771:84 , srv1171812867:84 , srv1175094958:84 , srv1242438855:84 , srv1271206825:84 , srv1332119282:84 , srv1334175522:84 , srv1335914510:84 , srv1346441738:84 , srv1355215298:84 , srv1367253421:84 , srv1481968388:84 , srv1499649765:84 , srv1540670536:84 , srv1568479659:84 , srv1608814280:84 , srv1616484850:84 , srv1620915649:84 , srv162210138:84 , srv1650437095:84 , srv1716535937:84 , srv1787454980:84 , srv1795441712:84 , srv1817138069:84 , srv1834796784:84 , srv1850707579:84 , srv188147830:84 , srv1894417345:84 , srv1904051029:84 , srv1914007702:84 , srv197060318:84 , srv2020323634:84 , srv216943000:84 , srv226561549:84 , srv239485881:84 , srv359050944:84 , srv573234305:84 , srv645356477:84 , srv726835124:84 , srv731375628:84 , srv732308977:84 , srv777562373:84 , srv787466352:84 , srv862572127:84 , srv923042643:84 , srv1022464852:85 , srv1030697889:85 , srv103328837:85 , srv1073378444:85 , srv1113533393:85 , srv1137393496:85 , srv1207214682:85 , srv120746057:85 , srv1235034004:85 , srv1244935290:85 , srv1303346323:85 , srv1326023051:85 , srv1372726230:85 , srv1389320164:85 , srv1403326166:85 , srv1469438706:85 , srv1473818121:85 , srv1509618917:85 , srv1586447321:85 , srv1644935070:85 , srv1722430794:85 , srv1809664420:85 , srv20578720:85 , srv2100639635:85 , srv210129325:85 , srv2124495009:85 , srv2125051468:85 , srv330543973:85 , srv499838464:85 , srv54115014:85 , srv552209071:85 , srv585798164:85 , srv60837499:85 , srv674335482:85 , srv715103551:85 , srv72585001:85 , srv779809382:85 , srv791659023:85 , srv820928841:85 , srv855554626:85 , srv879473460:85 , srv89972053:85 , srv608143639:86 , srv612891368:86 , srv663770124:86 , srv1424000280:644 }
2023-11-17 19:52:51,718 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 315), ProcessCount=175 (was 175), AvailableMemoryMB=3605 (was 4275)
2023-11-17 19:52:51,725 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=175, AvailableMemoryMB=3605
2023-11-17 19:52:51,726 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-17 19:52:51,726 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-17 19:52:51,735 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1142587108:84 , srv1243456146:84 , srv1337846299:84 , srv1391289817:84 , srv1409536448:84 , srv145329930:84 , srv1605589556:84 , srv1632365058:84 , srv1686878964:84 , srv1737177988:84 , srv1839066144:84 , srv1916708299:84 , srv1930589858:84 , srv2013813362:84 , srv2039952149:84 , srv2047476274:84 , srv228701689:84 , srv318088446:84 , srv322870537:84 , srv395580241:84 , srv445146254:84 , srv463890153:84 , srv505734413:84 , srv512019130:84 , srv577385336:84 , srv68241432:84 , srv715169798:84 , srv743401425:84 , srv859812134:84 , srv649987113:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-17 19:52:51,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:52:51,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv715169798=27, srv1142587108=0, srv322870537=18, srv743401425=28, srv512019130=23, srv1839066144=10, srv1916708299=11, srv2047476274=15, srv2013813362=13, srv2039952149=14, srv1632365058=7, srv395580241=19, srv577385336=24, srv145329930=5, srv649987113=25, srv1605589556=6, srv68241432=26, srv318088446=17, srv1409536448=4, srv228701689=16, srv859812134=29, srv505734413=22, srv1930589858=12, srv1391289817=3, srv463890153=21, srv1686878964=8, srv1243456146=1, srv1737177988=9, srv445146254=20, srv1337846299=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:52:51,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:52:51,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:52:51,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:52:51,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:52:51,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:52:51,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:52:51,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-17 19:52:51,760 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0862472336434181, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:52:51,760 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912464805481259, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0862472336434181, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-17 19:54:51,750 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9698185 different iterations.  Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912464805481259 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:54:51,755 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1142587108:90 , srv1243456146:90 , srv1337846299:90 , srv1391289817:90 , srv1409536448:90 , srv145329930:90 , srv1605589556:90 , srv1632365058:90 , srv1686878964:90 , srv1737177988:90 , srv1839066144:90 , srv1916708299:90 , srv1930589858:90 , srv2013813362:90 , srv2039952149:90 , srv2047476274:90 , srv228701689:90 , srv318088446:90 , srv322870537:90 , srv395580241:90 , srv445146254:90 , srv463890153:90 , srv505734413:90 , srv512019130:90 , srv577385336:90 , srv649987113:90 , srv68241432:90 , srv715169798:90 , srv743401425:90 , srv859812134:90 }
2023-11-17 19:54:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-17 19:54:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv715169798=27, srv1142587108=0, srv322870537=18, srv743401425=28, srv512019130=23, srv1839066144=10, srv1916708299=11, srv2047476274=15, srv2013813362=13, srv2039952149=14, srv1632365058=7, srv395580241=19, srv577385336=24, srv145329930=5, srv649987113=25, srv1605589556=6, srv68241432=26, srv318088446=17, srv1409536448=4, srv228701689=16, srv859812134=29, srv505734413=22, srv1930589858=12, srv1391289817=3, srv463890153=21, srv1686878964=8, srv1243456146=1, srv1737177988=9, srv445146254=20, srv1337846299=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-17 19:54:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-17 19:54:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-17 19:54:51,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-17 19:54:51,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-17 19:54:51,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-17 19:54:51,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-17 19:54:51,769 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-17 19:54:51,770 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1142587108:90 , srv1243456146:90 , srv1337846299:90 , srv1391289817:90 , srv1409536448:90 , srv145329930:90 , srv1605589556:90 , srv1632365058:90 , srv1686878964:90 , srv1737177988:90 , srv1839066144:90 , srv1916708299:90 , srv1930589858:90 , srv2013813362:90 , srv2039952149:90 , srv2047476274:90 , srv228701689:90 , srv318088446:90 , srv322870537:90 , srv395580241:90 , srv445146254:90 , srv463890153:90 , srv505734413:90 , srv512019130:90 , srv577385336:90 , srv649987113:90 , srv68241432:90 , srv715169798:90 , srv743401425:90 , srv859812134:90 }
2023-11-17 19:54:51,780 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=162 (was 315), ProcessCount=174 (was 175), AvailableMemoryMB=2755 (was 3605)