2024-01-22 00:53:16,446 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-22 00:53:16,788 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-22 00:53:16,909 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-22 00:53:16,973 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-22 00:53:16,980 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-22 00:53:16,994 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=167, AvailableMemoryMB=18824
2024-01-22 00:53:16,997 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-22 00:53:16,997 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-22 00:53:17,144 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100700301:84 , srv1017871249:84 , srv1047838573:84 , srv1061001147:84 , srv1087790793:84 , srv1125325815:84 , srv1160428307:84 , srv117312083:84 , srv1173897849:84 , srv1198393897:84 , srv1254955904:84 , srv1256348983:84 , srv1258759332:84 , srv1269011777:84 , srv132867129:84 , srv1339479033:84 , srv137003423:84 , srv1393097442:84 , srv1401219108:84 , srv1407432529:84 , srv1417428916:84 , srv1476996972:84 , srv1488285154:84 , srv1516330079:84 , srv1532278682:84 , srv154908537:84 , srv1552513127:84 , srv1570728310:84 , srv1573457666:84 , srv1582534264:84 , srv1601725223:84 , srv1605496167:84 , srv1611784786:84 , srv1617045909:84 , srv1635530903:84 , srv1647216596:84 , srv1665605100:84 , srv1672088081:84 , srv1684367674:84 , srv1826328442:84 , srv1852218204:84 , srv1899593030:84 , srv1927744441:84 , srv1954239434:84 , srv1986771311:84 , srv2080279123:84 , srv2084217402:84 , srv2089000768:84 , srv2110872484:84 , srv2125783891:84 , srv2134065437:84 , srv218706703:84 , srv223501099:84 , srv237900579:84 , srv273532008:84 , srv278251188:84 , srv280004718:84 , srv320724863:84 , srv335671762:84 , srv355923019:84 , srv356895224:84 , srv363375586:84 , srv372027563:84 , srv387742515:84 , srv400701150:84 , srv404830309:84 , srv407065939:84 , srv419077489:84 , srv419482154:84 , srv442227265:84 , srv472225878:84 , srv547547207:84 , srv561237893:84 , srv562421633:84 , srv576706603:84 , srv614103977:84 , srv615465301:84 , srv623177644:84 , srv635612882:84 , srv646793017:84 , srv662375588:84 , srv668195983:84 , srv670732837:84 , srv688160114:84 , srv69792576:84 , srv726065198:84 , srv748024376:84 , srv754655748:84 , srv75517652:84 , srv766512185:84 , srv770700961:84 , srv782788326:84 , srv823623367:84 , srv836998536:84 , srv844521712:84 , srv85869579:84 , srv869798232:84 , srv882524079:84 , srv91759821:84 , srv65501360:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-22 00:53:17,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1254955904=10, srv1582534264=29, srv670732837=83, srv356895224=60, srv1986771311=44, srv2089000768=47, srv782788326=92, srv2134065437=50, srv1061001147=3, srv869798232=97, srv280004718=56, srv1635530903=34, srv646793017=79, srv1417428916=20, srv355923019=59, srv100700301=0, srv273532008=54, srv562421633=73, srv1256348983=11, srv1826328442=39, srv117312083=7, srv1516330079=23, srv823623367=93, srv218706703=51, srv1258759332=12, srv1476996972=21, srv404830309=65, srv1407432529=19, srv614103977=75, srv770700961=91, srv1647216596=35, srv1087790793=4, srv335671762=58, srv748024376=87, srv1601725223=30, srv419482154=68, srv1173897849=8, srv1125325815=5, srv662375588=81, srv1532278682=24, srv472225878=70, srv2110872484=48, srv1047838573=2, srv561237893=72, srv1017871249=1, srv615465301=76, srv1852218204=40, srv69792576=85, srv400701150=64, srv137003423=16, srv1954239434=43, srv442227265=69, srv387742515=63, srv2084217402=46, srv766512185=90, srv320724863=57, srv668195983=82, srv1573457666=28, srv1611784786=32, srv688160114=84, srv1198393897=9, srv237900579=53, srv2125783891=49, srv635612882=78, srv1160428307=6, srv1552513127=26, srv91759821=99, srv726065198=86, srv1672088081=37, srv1269011777=13, srv419077489=67, srv1488285154=22, srv1393097442=17, srv1401219108=18, srv85869579=96, srv65501360=80, srv363375586=61, srv844521712=95, srv1684367674=38, srv154908537=25, srv1927744441=42, srv132867129=14, srv223501099=52, srv1605496167=31, srv754655748=88, srv1665605100=36, srv623177644=77, srv836998536=94, srv2080279123=45, srv1899593030=41, srv547547207=71, srv372027563=62, srv576706603=74, srv882524079=98, srv75517652=89, srv278251188=55, srv1570728310=27, srv1339479033=15, srv407065939=66, srv1617045909=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:17,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:17,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:17,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:17,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:17,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:17,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:17,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-22 00:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-22 00:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-22 00:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-22 00:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-22 00:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-22 00:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-22 00:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-22 00:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-22 00:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-22 00:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-22 00:53:17,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-22 00:53:17,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-22 00:53:17,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-22 00:53:17,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-22 00:53:17,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-22 00:53:17,448 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:17,449 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-22 00:53:17,449 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-22 00:53:18,245 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 942 ms to try 5000 different iterations.  Found a solution that moves 4659 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18653827349751412. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039820647916612495, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05703485062930839, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5176666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039820647916612495, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18216666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:18,271 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv419482154:81 , srv1256348983:82 , srv1516330079:83 , srv1826328442:83 , srv2080279123:83 , srv614103977:83 , srv646793017:83 , srv726065198:83 , srv823623367:83 , srv1160428307:84 , srv1173897849:84 , srv1339479033:84 , srv1476996972:84 , srv1552513127:84 , srv1635530903:84 , srv1684367674:84 , srv218706703:84 , srv280004718:84 , srv400701150:84 , srv407065939:84 , srv576706603:84 , srv670732837:84 , srv1047838573:85 , srv117312083:85 , srv1254955904:85 , srv1393097442:85 , srv1582534264:85 , srv1927744441:85 , srv273532008:85 , srv356895224:85 , srv372027563:85 , srv387742515:85 , srv419077489:85 , srv662375588:85 , srv782788326:85 , srv836998536:85 , srv869798232:85 , srv1198393897:86 , srv132867129:86 , srv137003423:86 , srv1401219108:86 , srv1407432529:86 , srv1488285154:86 , srv1617045909:86 , srv1899593030:86 , srv1986771311:86 , srv237900579:86 , srv363375586:86 , srv442227265:86 , srv668195983:86 , srv688160114:86 , srv69792576:86 , srv85869579:86 , srv882524079:86 , srv91759821:86 , srv1017871249:87 , srv1258759332:87 , srv1269011777:87 , srv1532278682:87 , srv1570728310:87 , srv1601725223:87 , srv1611784786:87 , srv1665605100:87 , srv1672088081:87 , srv2084217402:87 , srv2110872484:87 , srv2134065437:87 , srv355923019:87 , srv75517652:87 , srv844521712:87 , srv1061001147:88 , srv1087790793:88 , srv1417428916:88 , srv154908537:88 , srv1647216596:88 , srv278251188:88 , srv320724863:88 , srv335671762:88 , srv472225878:88 , srv561237893:88 , srv623177644:88 , srv748024376:88 , srv770700961:88 , srv100700301:89 , srv1954239434:89 , srv2125783891:89 , srv754655748:89 , srv1605496167:90 , srv1852218204:90 , srv404830309:90 , srv547547207:90 , srv562421633:90 , srv615465301:90 , srv1125325815:91 , srv1573457666:91 , srv2089000768:92 , srv223501099:92 , srv766512185:92 , srv635612882:94 , srv65501360:444 }
2024-01-22 00:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:18,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1254955904=10, srv1582534264=29, srv670732837=83, srv356895224=60, srv1986771311=44, srv2089000768=47, srv782788326=92, srv2134065437=50, srv1061001147=3, srv869798232=97, srv280004718=56, srv1635530903=34, srv646793017=79, srv1417428916=20, srv355923019=59, srv100700301=0, srv273532008=54, srv562421633=73, srv1256348983=11, srv1826328442=39, srv117312083=7, srv1516330079=23, srv823623367=93, srv218706703=51, srv1258759332=12, srv1476996972=21, srv404830309=65, srv1407432529=19, srv614103977=75, srv770700961=91, srv1647216596=35, srv1087790793=4, srv335671762=58, srv748024376=87, srv1601725223=30, srv419482154=68, srv1173897849=8, srv1125325815=5, srv662375588=81, srv1532278682=24, srv472225878=70, srv2110872484=48, srv1047838573=2, srv561237893=72, srv1017871249=1, srv615465301=76, srv1852218204=40, srv69792576=85, srv400701150=64, srv137003423=16, srv1954239434=43, srv442227265=69, srv387742515=63, srv2084217402=46, srv766512185=90, srv320724863=57, srv668195983=82, srv1573457666=28, srv1611784786=32, srv688160114=84, srv1198393897=9, srv237900579=53, srv2125783891=49, srv635612882=78, srv1160428307=6, srv1552513127=26, srv91759821=99, srv726065198=86, srv1672088081=37, srv1269011777=13, srv419077489=67, srv1488285154=22, srv1393097442=17, srv1401219108=18, srv85869579=96, srv65501360=80, srv363375586=61, srv844521712=95, srv1684367674=38, srv154908537=25, srv1927744441=42, srv132867129=14, srv223501099=52, srv1605496167=31, srv754655748=88, srv1665605100=36, srv623177644=77, srv836998536=94, srv2080279123=45, srv1899593030=41, srv547547207=71, srv372027563=62, srv576706603=74, srv882524079=98, srv75517652=89, srv278251188=55, srv1570728310=27, srv1339479033=15, srv407065939=66, srv1617045909=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:18,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:18,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-22 00:53:18,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-22 00:53:18,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-22 00:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-22 00:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-22 00:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-22 00:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-22 00:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-22 00:53:18,457 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039820647916612495, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05703485062930839, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039820647916612495, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18216666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:18,457 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-22 00:53:18,458 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18650565048224846, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039820647916612495, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05703485062930839, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039820647916612495, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18216666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.245, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-22 00:53:18,745 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 410 ms to try 5000 different iterations.  Found a solution that moves 2663 regions; Going from a computed imbalance of 0.18650565048224846 to a new imbalance of 0.015108845689026162. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020936426554034368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03327941767598968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29588888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020936426554034368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:18,753 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv766512185:79 , srv1125325815:80 , srv2089000768:80 , srv635612882:80 , srv1573457666:81 , srv1605496167:81 , srv223501099:81 , srv547547207:82 , srv770700961:82 , srv154908537:83 , srv1570728310:83 , srv1852218204:83 , srv1954239434:83 , srv278251188:83 , srv472225878:83 , srv562421633:83 , srv754655748:83 , srv91759821:83 , srv1061001147:84 , srv1087790793:84 , srv117312083:84 , srv1269011777:84 , srv1611784786:84 , srv1647216596:84 , srv1986771311:84 , srv2125783891:84 , srv273532008:84 , srv320724863:84 , srv355923019:84 , srv404830309:84 , srv615465301:84 , srv623177644:84 , srv688160114:84 , srv69792576:84 , srv748024376:84 , srv75517652:84 , srv100700301:85 , srv137003423:85 , srv1617045909:85 , srv2110872484:85 , srv2134065437:85 , srv218706703:85 , srv335671762:85 , srv561237893:85 , srv782788326:85 , srv844521712:85 , srv1198393897:86 , srv1254955904:86 , srv132867129:86 , srv1393097442:86 , srv1407432529:86 , srv1417428916:86 , srv1476996972:86 , srv1532278682:86 , srv1601725223:86 , srv1665605100:86 , srv1672088081:86 , srv2084217402:86 , srv280004718:86 , srv356895224:86 , srv363375586:86 , srv882524079:86 , srv1017871249:87 , srv1173897849:87 , srv1258759332:87 , srv1684367674:87 , srv387742515:87 , srv419077489:87 , srv614103977:87 , srv836998536:87 , srv85869579:87 , srv869798232:87 , srv1047838573:88 , srv1401219108:88 , srv1488285154:88 , srv1582534264:88 , srv1635530903:88 , srv1899593030:88 , srv1927744441:88 , srv237900579:88 , srv400701150:88 , srv576706603:88 , srv662375588:88 , srv670732837:88 , srv1160428307:89 , srv1516330079:89 , srv1552513127:89 , srv1826328442:89 , srv372027563:89 , srv442227265:89 , srv646793017:89 , srv668195983:89 , srv823623367:89 , srv1256348983:90 , srv1339479033:91 , srv2080279123:91 , srv419482154:91 , srv726065198:91 , srv407065939:92 , srv65501360:516 }
2024-01-22 00:53:18,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:18,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1254955904=10, srv1582534264=29, srv670732837=83, srv356895224=60, srv1986771311=44, srv2089000768=47, srv782788326=92, srv2134065437=50, srv1061001147=3, srv869798232=97, srv280004718=56, srv1635530903=34, srv646793017=79, srv1417428916=20, srv355923019=59, srv100700301=0, srv273532008=54, srv562421633=73, srv1256348983=11, srv1826328442=39, srv117312083=7, srv1516330079=23, srv823623367=93, srv218706703=51, srv1258759332=12, srv1476996972=21, srv404830309=65, srv1407432529=19, srv614103977=75, srv770700961=91, srv1647216596=35, srv1087790793=4, srv335671762=58, srv748024376=87, srv1601725223=30, srv419482154=68, srv1173897849=8, srv1125325815=5, srv662375588=81, srv1532278682=24, srv472225878=70, srv2110872484=48, srv1047838573=2, srv561237893=72, srv1017871249=1, srv615465301=76, srv1852218204=40, srv69792576=85, srv400701150=64, srv137003423=16, srv1954239434=43, srv442227265=69, srv387742515=63, srv2084217402=46, srv766512185=90, srv320724863=57, srv668195983=82, srv1573457666=28, srv1611784786=32, srv688160114=84, srv1198393897=9, srv237900579=53, srv2125783891=49, srv635612882=78, srv1160428307=6, srv1552513127=26, srv91759821=99, srv726065198=86, srv1672088081=37, srv1269011777=13, srv419077489=67, srv1488285154=22, srv1393097442=17, srv1401219108=18, srv85869579=96, srv65501360=80, srv363375586=61, srv844521712=95, srv1684367674=38, srv154908537=25, srv1927744441=42, srv132867129=14, srv223501099=52, srv1605496167=31, srv754655748=88, srv1665605100=36, srv623177644=77, srv836998536=94, srv2080279123=45, srv1899593030=41, srv547547207=71, srv372027563=62, srv576706603=74, srv882524079=98, srv75517652=89, srv278251188=55, srv1570728310=27, srv1339479033=15, srv407065939=66, srv1617045909=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-22 00:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-22 00:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-22 00:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-22 00:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-22 00:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-22 00:53:18,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020936426554034368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03327941767598968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020936426554034368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:18,890 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-22 00:53:18,890 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015090198964481726, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020936426554034368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03327941767598968, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020936426554034368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-22 00:53:19,115 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 326 ms to try 5000 different iterations.  Found a solution that moves 1014 regions; Going from a computed imbalance of 0.015090198964481726 to a new imbalance of 4.146849843021662E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009247804362219473); MoveCostFunction : (multiplier=7.0, imbalance=0.11266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:19,117 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv407065939:81 , srv1899593030:82 , srv2080279123:82 , srv668195983:82 , srv100700301:83 , srv1017871249:83 , srv1258759332:83 , srv1339479033:83 , srv1417428916:83 , srv1488285154:83 , srv237900579:83 , srv372027563:83 , srv404830309:83 , srv442227265:83 , srv726065198:83 , srv1160428307:84 , srv1198393897:84 , srv1401219108:84 , srv1532278682:84 , srv1601725223:84 , srv1665605100:84 , srv1672088081:84 , srv1852218204:84 , srv2084217402:84 , srv223501099:84 , srv400701150:84 , srv561237893:84 , srv615465301:84 , srv635612882:84 , srv754655748:84 , srv85869579:84 , srv1061001147:85 , srv1269011777:85 , srv1516330079:85 , srv1552513127:85 , srv1582534264:85 , srv1826328442:85 , srv1927744441:85 , srv1954239434:85 , srv2110872484:85 , srv2125783891:85 , srv2134065437:85 , srv320724863:85 , srv387742515:85 , srv419077489:85 , srv419482154:85 , srv547547207:85 , srv562421633:85 , srv576706603:85 , srv623177644:85 , srv646793017:85 , srv662375588:85 , srv748024376:85 , srv836998536:85 , srv869798232:85 , srv882524079:85 , srv1047838573:86 , srv1087790793:86 , srv1125325815:86 , srv1254955904:86 , srv1256348983:86 , srv132867129:86 , srv137003423:86 , srv1407432529:86 , srv154908537:86 , srv1573457666:86 , srv1605496167:86 , srv1617045909:86 , srv1635530903:86 , srv1647216596:86 , srv1684367674:86 , srv278251188:86 , srv335671762:86 , srv363375586:86 , srv472225878:86 , srv823623367:86 , srv1173897849:87 , srv1393097442:87 , srv1476996972:87 , srv1570728310:87 , srv1611784786:87 , srv1986771311:87 , srv2089000768:87 , srv280004718:87 , srv355923019:87 , srv356895224:87 , srv614103977:87 , srv670732837:87 , srv688160114:87 , srv69792576:87 , srv75517652:87 , srv766512185:87 , srv782788326:87 , srv844521712:87 , srv117312083:88 , srv218706703:88 , srv770700961:88 , srv91759821:88 , srv273532008:89 , srv65501360:564 }
2024-01-22 00:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1254955904=10, srv1582534264=29, srv670732837=83, srv356895224=60, srv1986771311=44, srv2089000768=47, srv782788326=92, srv2134065437=50, srv1061001147=3, srv869798232=97, srv280004718=56, srv1635530903=34, srv646793017=79, srv1417428916=20, srv355923019=59, srv100700301=0, srv273532008=54, srv562421633=73, srv1256348983=11, srv1826328442=39, srv117312083=7, srv1516330079=23, srv823623367=93, srv218706703=51, srv1258759332=12, srv1476996972=21, srv404830309=65, srv1407432529=19, srv614103977=75, srv770700961=91, srv1647216596=35, srv1087790793=4, srv335671762=58, srv748024376=87, srv1601725223=30, srv419482154=68, srv1173897849=8, srv1125325815=5, srv662375588=81, srv1532278682=24, srv472225878=70, srv2110872484=48, srv1047838573=2, srv561237893=72, srv1017871249=1, srv615465301=76, srv1852218204=40, srv69792576=85, srv400701150=64, srv137003423=16, srv1954239434=43, srv442227265=69, srv387742515=63, srv2084217402=46, srv766512185=90, srv320724863=57, srv668195983=82, srv1573457666=28, srv1611784786=32, srv688160114=84, srv1198393897=9, srv237900579=53, srv2125783891=49, srv635612882=78, srv1160428307=6, srv1552513127=26, srv91759821=99, srv726065198=86, srv1672088081=37, srv1269011777=13, srv419077489=67, srv1488285154=22, srv1393097442=17, srv1401219108=18, srv85869579=96, srv65501360=80, srv363375586=61, srv844521712=95, srv1684367674=38, srv154908537=25, srv1927744441=42, srv132867129=14, srv223501099=52, srv1605496167=31, srv754655748=88, srv1665605100=36, srv623177644=77, srv836998536=94, srv2080279123=45, srv1899593030=41, srv547547207=71, srv372027563=62, srv576706603=74, srv882524079=98, srv75517652=89, srv278251188=55, srv1570728310=27, srv1339479033=15, srv407065939=66, srv1617045909=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:19,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:19,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:19,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:19,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:19,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-22 00:53:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-22 00:53:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-22 00:53:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-22 00:53:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-22 00:53:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-22 00:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-22 00:53:19,251 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009247804362219473); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:19,251 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-22 00:53:19,251 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.0758480454698134E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007444352311941399); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009247804362219473); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007444352311941399); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-22 00:53:19,580 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 427 ms to try 5000 different iterations.  Found a solution that moves 257 regions; Going from a computed imbalance of 4.0758480454698134E-4 to a new imbalance of 1.9843567878791523E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:19,581 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv356895224:82 , srv1047838573:83 , srv1087790793:83 , srv2089000768:83 , srv335671762:83 , srv670732837:83 , srv117312083:84 , srv1173897849:84 , srv1256348983:84 , srv1339479033:84 , srv1393097442:84 , srv1407432529:84 , srv1417428916:84 , srv1552513127:84 , srv1573457666:84 , srv1582534264:84 , srv1611784786:84 , srv1635530903:84 , srv1647216596:84 , srv1665605100:84 , srv1927744441:84 , srv2110872484:84 , srv2125783891:84 , srv273532008:84 , srv355923019:84 , srv363375586:84 , srv561237893:84 , srv562421633:84 , srv615465301:84 , srv635612882:84 , srv75517652:84 , srv766512185:84 , srv770700961:84 , srv823623367:84 , srv844521712:84 , srv100700301:85 , srv1017871249:85 , srv1061001147:85 , srv1125325815:85 , srv1160428307:85 , srv1254955904:85 , srv1258759332:85 , srv132867129:85 , srv137003423:85 , srv1401219108:85 , srv1476996972:85 , srv1488285154:85 , srv1516330079:85 , srv1532278682:85 , srv154908537:85 , srv1570728310:85 , srv1601725223:85 , srv1605496167:85 , srv1617045909:85 , srv1672088081:85 , srv1684367674:85 , srv1826328442:85 , srv1852218204:85 , srv1954239434:85 , srv1986771311:85 , srv2084217402:85 , srv2134065437:85 , srv223501099:85 , srv237900579:85 , srv278251188:85 , srv280004718:85 , srv320724863:85 , srv372027563:85 , srv387742515:85 , srv404830309:85 , srv407065939:85 , srv419077489:85 , srv419482154:85 , srv442227265:85 , srv472225878:85 , srv547547207:85 , srv576706603:85 , srv614103977:85 , srv623177644:85 , srv646793017:85 , srv662375588:85 , srv688160114:85 , srv69792576:85 , srv726065198:85 , srv748024376:85 , srv782788326:85 , srv836998536:85 , srv85869579:85 , srv869798232:85 , srv882524079:85 , srv91759821:85 , srv1198393897:86 , srv1269011777:86 , srv1899593030:86 , srv218706703:86 , srv400701150:86 , srv668195983:86 , srv754655748:86 , srv2080279123:87 , srv65501360:618 }
2024-01-22 00:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:19,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1254955904=10, srv1582534264=29, srv670732837=83, srv356895224=60, srv1986771311=44, srv2089000768=47, srv782788326=92, srv2134065437=50, srv1061001147=3, srv869798232=97, srv280004718=56, srv1635530903=34, srv646793017=79, srv1417428916=20, srv355923019=59, srv100700301=0, srv273532008=54, srv562421633=73, srv1256348983=11, srv1826328442=39, srv117312083=7, srv1516330079=23, srv823623367=93, srv218706703=51, srv1258759332=12, srv1476996972=21, srv404830309=65, srv1407432529=19, srv614103977=75, srv770700961=91, srv1647216596=35, srv1087790793=4, srv335671762=58, srv748024376=87, srv1601725223=30, srv419482154=68, srv1173897849=8, srv1125325815=5, srv662375588=81, srv1532278682=24, srv472225878=70, srv2110872484=48, srv1047838573=2, srv561237893=72, srv1017871249=1, srv615465301=76, srv1852218204=40, srv69792576=85, srv400701150=64, srv137003423=16, srv1954239434=43, srv442227265=69, srv387742515=63, srv2084217402=46, srv766512185=90, srv320724863=57, srv668195983=82, srv1573457666=28, srv1611784786=32, srv688160114=84, srv1198393897=9, srv237900579=53, srv2125783891=49, srv635612882=78, srv1160428307=6, srv1552513127=26, srv91759821=99, srv726065198=86, srv1672088081=37, srv1269011777=13, srv419077489=67, srv1488285154=22, srv1393097442=17, srv1401219108=18, srv85869579=96, srv65501360=80, srv363375586=61, srv844521712=95, srv1684367674=38, srv154908537=25, srv1927744441=42, srv132867129=14, srv223501099=52, srv1605496167=31, srv754655748=88, srv1665605100=36, srv623177644=77, srv836998536=94, srv2080279123=45, srv1899593030=41, srv547547207=71, srv372027563=62, srv576706603=74, srv882524079=98, srv75517652=89, srv278251188=55, srv1570728310=27, srv1339479033=15, srv407065939=66, srv1617045909=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:19,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:19,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-22 00:53:19,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-22 00:53:19,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-22 00:53:19,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-22 00:53:19,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-22 00:53:19,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-22 00:53:19,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-22 00:53:19,705 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8044015416185505E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:19,706 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv356895224:82 , srv1047838573:83 , srv1087790793:83 , srv2089000768:83 , srv335671762:83 , srv670732837:83 , srv117312083:84 , srv1173897849:84 , srv1256348983:84 , srv1339479033:84 , srv1393097442:84 , srv1407432529:84 , srv1417428916:84 , srv1552513127:84 , srv1573457666:84 , srv1582534264:84 , srv1611784786:84 , srv1635530903:84 , srv1647216596:84 , srv1665605100:84 , srv1927744441:84 , srv2110872484:84 , srv2125783891:84 , srv273532008:84 , srv355923019:84 , srv363375586:84 , srv561237893:84 , srv562421633:84 , srv615465301:84 , srv635612882:84 , srv75517652:84 , srv766512185:84 , srv770700961:84 , srv823623367:84 , srv844521712:84 , srv100700301:85 , srv1017871249:85 , srv1061001147:85 , srv1125325815:85 , srv1160428307:85 , srv1254955904:85 , srv1258759332:85 , srv132867129:85 , srv137003423:85 , srv1401219108:85 , srv1476996972:85 , srv1488285154:85 , srv1516330079:85 , srv1532278682:85 , srv154908537:85 , srv1570728310:85 , srv1601725223:85 , srv1605496167:85 , srv1617045909:85 , srv1672088081:85 , srv1684367674:85 , srv1826328442:85 , srv1852218204:85 , srv1954239434:85 , srv1986771311:85 , srv2084217402:85 , srv2134065437:85 , srv223501099:85 , srv237900579:85 , srv278251188:85 , srv280004718:85 , srv320724863:85 , srv372027563:85 , srv387742515:85 , srv404830309:85 , srv407065939:85 , srv419077489:85 , srv419482154:85 , srv442227265:85 , srv472225878:85 , srv547547207:85 , srv576706603:85 , srv614103977:85 , srv623177644:85 , srv646793017:85 , srv662375588:85 , srv688160114:85 , srv69792576:85 , srv726065198:85 , srv748024376:85 , srv782788326:85 , srv836998536:85 , srv85869579:85 , srv869798232:85 , srv882524079:85 , srv91759821:85 , srv1198393897:86 , srv1269011777:86 , srv1899593030:86 , srv218706703:86 , srv400701150:86 , srv668195983:86 , srv754655748:86 , srv2080279123:87 , srv65501360:618 }
2024-01-22 00:53:19,731 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=285 (was 275) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=18234 (was 18824)
2024-01-22 00:53:19,739 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=167, AvailableMemoryMB=18233
2024-01-22 00:53:19,739 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-22 00:53:19,740 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-22 00:53:19,753 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010715430:84 , srv1018603970:84 , srv1274503630:84 , srv1284205220:84 , srv1319256551:84 , srv1391489082:84 , srv1594351296:84 , srv1648164925:84 , srv1653231814:84 , srv1823601646:84 , srv1855110498:84 , srv1869056122:84 , srv1900424795:84 , srv190388525:84 , srv1982347644:84 , srv2074349572:84 , srv209390750:84 , srv340479959:84 , srv424813596:84 , srv442437625:84 , srv46109028:84 , srv514067677:84 , srv66515914:84 , srv699561445:84 , srv802175742:84 , srv80444727:84 , srv822359822:84 , srv859933396:84 , srv997256612:84 , srv1902734858:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-22 00:53:19,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:53:19,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv442437625=20, srv514067677=22, srv997256612=29, srv190388525=14, srv802175742=25, srv1284205220=3, srv46109028=21, srv1653231814=8, srv699561445=24, srv66515914=23, srv822359822=27, srv2074349572=16, srv1319256551=4, srv1855110498=10, srv1900424795=12, srv1594351296=6, srv859933396=28, srv1823601646=9, srv1648164925=7, srv1018603970=1, srv424813596=19, srv80444727=26, srv1274503630=2, srv1391489082=5, srv209390750=17, srv340479959=18, srv1869056122=11, srv1010715430=0, srv1902734858=13, srv1982347644=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:53:19,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:53:19,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:53:19,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:53:19,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-22 00:53:19,801 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0739861648765996, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:53:19,801 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912426171253935, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0739861648765996, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-22 00:55:19,776 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6268324 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912426171253935 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:55:19,782 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010715430:90 , srv1018603970:90 , srv1274503630:90 , srv1284205220:90 , srv1319256551:90 , srv1391489082:90 , srv1594351296:90 , srv1648164925:90 , srv1653231814:90 , srv1823601646:90 , srv1855110498:90 , srv1869056122:90 , srv1900424795:90 , srv1902734858:90 , srv190388525:90 , srv1982347644:90 , srv2074349572:90 , srv209390750:90 , srv340479959:90 , srv424813596:90 , srv442437625:90 , srv46109028:90 , srv514067677:90 , srv66515914:90 , srv699561445:90 , srv802175742:90 , srv80444727:90 , srv822359822:90 , srv859933396:90 , srv997256612:90 }
2024-01-22 00:55:19,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-22 00:55:19,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv442437625=20, srv514067677=22, srv997256612=29, srv190388525=14, srv802175742=25, srv1284205220=3, srv46109028=21, srv1653231814=8, srv699561445=24, srv66515914=23, srv822359822=27, srv2074349572=16, srv1319256551=4, srv1855110498=10, srv1900424795=12, srv1594351296=6, srv859933396=28, srv1823601646=9, srv1648164925=7, srv1018603970=1, srv424813596=19, srv80444727=26, srv1274503630=2, srv1391489082=5, srv209390750=17, srv340479959=18, srv1869056122=11, srv1010715430=0, srv1902734858=13, srv1982347644=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-22 00:55:19,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-22 00:55:19,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-22 00:55:19,799 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-22 00:55:19,799 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1010715430:90 , srv1018603970:90 , srv1274503630:90 , srv1284205220:90 , srv1319256551:90 , srv1391489082:90 , srv1594351296:90 , srv1648164925:90 , srv1653231814:90 , srv1823601646:90 , srv1855110498:90 , srv1869056122:90 , srv1900424795:90 , srv1902734858:90 , srv190388525:90 , srv1982347644:90 , srv2074349572:90 , srv209390750:90 , srv340479959:90 , srv424813596:90 , srv442437625:90 , srv46109028:90 , srv514067677:90 , srv66515914:90 , srv699561445:90 , srv802175742:90 , srv80444727:90 , srv822359822:90 , srv859933396:90 , srv997256612:90 }
2024-01-22 00:55:19,812 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=249 (was 285), ProcessCount=167 (was 167), AvailableMemoryMB=17509 (was 18233)