2024-01-03 15:53:17,483 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-03 15:53:17,802 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-03 15:53:17,921 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-03 15:53:17,987 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 15:53:17,994 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-03 15:53:18,008 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=167, AvailableMemoryMB=4464
2024-01-03 15:53:18,011 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 15:53:18,011 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-03 15:53:18,140 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100843619:84 , srv1027407816:84 , srv1044487562:84 , srv1065818905:84 , srv1084126698:84 , srv1085396864:84 , srv1093820189:84 , srv1116220391:84 , srv1137842997:84 , srv1145338614:84 , srv1149388564:84 , srv1171399308:84 , srv1171832959:84 , srv1214920855:84 , srv1265840734:84 , srv1275927246:84 , srv1307965577:84 , srv1326326719:84 , srv1337860894:84 , srv1357284277:84 , srv1362887715:84 , srv1364624467:84 , srv1415280477:84 , srv1442929482:84 , srv1446237579:84 , srv1449132190:84 , srv1452234545:84 , srv1455653067:84 , srv1472052947:84 , srv1501771116:84 , srv1523496837:84 , srv1540580881:84 , srv1561150847:84 , srv1566457530:84 , srv156805258:84 , srv1577495072:84 , srv1590007611:84 , srv1592811339:84 , srv1615138127:84 , srv1626704118:84 , srv1637997828:84 , srv1652376265:84 , srv1662188701:84 , srv1663923836:84 , srv1689771928:84 , srv1721019398:84 , srv1734921054:84 , srv1776449538:84 , srv1778177653:84 , srv1796264790:84 , srv1838702572:84 , srv183932182:84 , srv1851850379:84 , srv1852245428:84 , srv1876774894:84 , srv1879181601:84 , srv18912068:84 , srv1918613391:84 , srv1918977088:84 , srv1926782625:84 , srv1934333882:84 , srv1940332414:84 , srv196231254:84 , srv1981222682:84 , srv2012981093:84 , srv201567571:84 , srv2031012284:84 , srv2054252687:84 , srv2104159603:84 , srv2106334971:84 , srv21542919:84 , srv301250672:84 , srv349343782:84 , srv349512453:84 , srv363017103:84 , srv364430792:84 , srv440020476:84 , srv454448893:84 , srv465723847:84 , srv502480498:84 , srv529576669:84 , srv548699470:84 , srv576378557:84 , srv639445801:84 , srv647302983:84 , srv66106739:84 , srv679820235:84 , srv683232945:84 , srv699089043:84 , srv737933936:84 , srv789772336:84 , srv792964562:84 , srv837699534:84 , srv844640991:84 , srv968823177:84 , srv976110566:84 , srv991057595:84 , srv993520350:84 , srv999487865:84 , srv1279654983:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-03 15:53:18,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:18,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:18,512 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:18,513 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 15:53:18,513 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 15:53:18,884 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 604 ms to try 5000 different iterations.  Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17697204712768924. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04073308774156723, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05581153149896875, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04073308774156723, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:18,896 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1085396864:82 , srv1093820189:82 , srv1149388564:82 , srv1455653067:82 , srv1615138127:82 , srv1778177653:82 , srv1918977088:82 , srv1362887715:83 , srv1364624467:83 , srv1449132190:83 , srv1472052947:83 , srv1523496837:83 , srv2031012284:83 , srv844640991:83 , srv968823177:83 , srv1415280477:84 , srv1689771928:84 , srv1734921054:84 , srv2054252687:84 , srv349343782:84 , srv440020476:84 , srv976110566:84 , srv999487865:84 , srv100843619:85 , srv1116220391:85 , srv1214920855:85 , srv1265840734:85 , srv1561150847:85 , srv1592811339:85 , srv1662188701:85 , srv1721019398:85 , srv1776449538:85 , srv1838702572:85 , srv1879181601:85 , srv454448893:85 , srv679820235:85 , srv699089043:85 , srv837699534:85 , srv1307965577:86 , srv1626704118:86 , srv1663923836:86 , srv1851850379:86 , srv1918613391:86 , srv1926782625:86 , srv1934333882:86 , srv1940332414:86 , srv201567571:86 , srv2104159603:86 , srv21542919:86 , srv349512453:86 , srv363017103:86 , srv465723847:86 , srv529576669:86 , srv66106739:86 , srv683232945:86 , srv737933936:86 , srv991057595:86 , srv1027407816:87 , srv1137842997:87 , srv1171832959:87 , srv1275927246:87 , srv1326326719:87 , srv1337860894:87 , srv1357284277:87 , srv1637997828:87 , srv1652376265:87 , srv1981222682:87 , srv301250672:87 , srv792964562:87 , srv1065818905:88 , srv1084126698:88 , srv1171399308:88 , srv1577495072:88 , srv1590007611:88 , srv183932182:88 , srv1876774894:88 , srv2012981093:88 , srv2106334971:88 , srv548699470:88 , srv993520350:88 , srv1044487562:89 , srv1442929482:89 , srv1452234545:89 , srv1566457530:89 , srv156805258:89 , srv502480498:89 , srv576378557:89 , srv647302983:89 , srv1446237579:90 , srv1540580881:90 , srv196231254:90 , srv789772336:90 , srv1501771116:91 , srv1796264790:91 , srv18912068:91 , srv1145338614:93 , srv1852245428:93 , srv364430792:93 , srv639445801:93 , srv1279654983:452 }
2024-01-03 15:53:18,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:18,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:18,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:18,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:18,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:18,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:18,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:18,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:18,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:18,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:19,052 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04073308774156723, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05581153149896875, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04073308774156723, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:19,052 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 15:53:19,053 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17693910201353918, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04073308774156723, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05581153149896875, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04073308774156723, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 15:53:19,312 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 374 ms to try 5000 different iterations.  Found a solution that moves 2595 regions; Going from a computed imbalance of 0.17693910201353918 to a new imbalance of 0.016521033893797006. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021500630035111098); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03483808950501085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021500630035111098); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:19,325 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1796264790:79 , srv364430792:79 , srv639445801:79 , srv1145338614:80 , srv576378557:80 , srv1501771116:81 , srv1577495072:81 , srv1852245428:81 , srv789772336:81 , srv1065818905:82 , srv1337860894:82 , srv502480498:82 , srv1084126698:83 , srv1137842997:83 , srv1171399308:83 , srv1442929482:83 , srv1540580881:83 , srv1566457530:83 , srv18912068:83 , srv2106334971:83 , srv647302983:83 , srv792964562:83 , srv1452234545:84 , srv156805258:84 , srv1626704118:84 , srv1876774894:84 , srv1981222682:84 , srv529576669:84 , srv679820235:84 , srv683232945:84 , srv1027407816:85 , srv1275927246:85 , srv1326326719:85 , srv1357284277:85 , srv1446237579:85 , srv1590007611:85 , srv1637997828:85 , srv1662188701:85 , srv1838702572:85 , srv183932182:85 , srv1851850379:85 , srv1934333882:85 , srv196231254:85 , srv2012981093:85 , srv465723847:85 , srv548699470:85 , srv993520350:85 , srv1044487562:86 , srv1171832959:86 , srv1214920855:86 , srv1265840734:86 , srv1307965577:86 , srv1663923836:86 , srv1721019398:86 , srv1776449538:86 , srv1918613391:86 , srv1926782625:86 , srv1940332414:86 , srv21542919:86 , srv301250672:86 , srv1561150847:87 , srv1615138127:87 , srv1652376265:87 , srv2054252687:87 , srv440020476:87 , srv699089043:87 , srv837699534:87 , srv976110566:87 , srv991057595:87 , srv1116220391:88 , srv1362887715:88 , srv1523496837:88 , srv1689771928:88 , srv1879181601:88 , srv2104159603:88 , srv349343782:88 , srv349512453:88 , srv363017103:88 , srv737933936:88 , srv968823177:88 , srv100843619:89 , srv1415280477:89 , srv1472052947:89 , srv1592811339:89 , srv1734921054:89 , srv1778177653:89 , srv2031012284:89 , srv454448893:89 , srv66106739:89 , srv844640991:89 , srv999487865:89 , srv1149388564:90 , srv1364624467:90 , srv1455653067:90 , srv201567571:90 , srv1085396864:91 , srv1093820189:91 , srv1449132190:91 , srv1918977088:92 , srv1279654983:513 }
2024-01-03 15:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:19,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:19,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:19,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:19,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:19,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:19,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:19,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:19,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:19,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:19,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:19,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:19,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:19,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021500630035111098); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03483808950501085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021500630035111098); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:19,486 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 15:53:19,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016502863315429446, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021500630035111098); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03483808950501085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021500630035111098); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 15:53:19,711 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 326 ms to try 5000 different iterations.  Found a solution that moves 1016 regions; Going from a computed imbalance of 0.016502863315429446 to a new imbalance of 8.910426326908418E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795712); MoveCostFunction : (multiplier=7.0, imbalance=0.11288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:19,713 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv201567571:82 , srv1044487562:83 , srv1446237579:83 , srv1592811339:83 , srv196231254:83 , srv2104159603:83 , srv363017103:83 , srv66106739:83 , srv737933936:83 , srv100843619:84 , srv1116220391:84 , srv1145338614:84 , srv1171832959:84 , srv1364624467:84 , srv1415280477:84 , srv1449132190:84 , srv1452234545:84 , srv1540580881:84 , srv156805258:84 , srv1652376265:84 , srv1734921054:84 , srv183932182:84 , srv1879181601:84 , srv18912068:84 , srv1918977088:84 , srv301250672:84 , srv454448893:84 , srv548699470:84 , srv999487865:84 , srv1027407816:85 , srv1085396864:85 , srv1093820189:85 , srv1149388564:85 , srv1275927246:85 , srv1326326719:85 , srv1357284277:85 , srv1442929482:85 , srv1455653067:85 , srv1472052947:85 , srv1501771116:85 , srv1566457530:85 , srv1590007611:85 , srv1637997828:85 , srv1663923836:85 , srv1926782625:85 , srv1940332414:85 , srv2012981093:85 , srv21542919:85 , srv349343782:85 , srv349512453:85 , srv364430792:85 , srv991057595:85 , srv993520350:85 , srv1265840734:86 , srv1307965577:86 , srv1362887715:86 , srv1523496837:86 , srv1561150847:86 , srv1689771928:86 , srv1721019398:86 , srv1776449538:86 , srv1851850379:86 , srv1852245428:86 , srv1876774894:86 , srv1918613391:86 , srv1981222682:86 , srv2031012284:86 , srv2054252687:86 , srv639445801:86 , srv647302983:86 , srv699089043:86 , srv837699534:86 , srv844640991:86 , srv976110566:86 , srv1084126698:87 , srv1137842997:87 , srv1171399308:87 , srv1214920855:87 , srv1626704118:87 , srv1778177653:87 , srv1838702572:87 , srv1934333882:87 , srv2106334971:87 , srv440020476:87 , srv465723847:87 , srv502480498:87 , srv789772336:87 , srv792964562:87 , srv968823177:87 , srv1065818905:88 , srv1337860894:88 , srv1577495072:88 , srv1796264790:88 , srv529576669:88 , srv679820235:88 , srv683232945:88 , srv1615138127:89 , srv576378557:89 , srv1662188701:90 , srv1279654983:539 }
2024-01-03 15:53:19,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:19,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:19,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:19,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:19,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:19,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:19,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:19,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:19,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:19,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:19,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:19,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:19,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:19,850 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795712); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:19,850 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 15:53:19,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.839284486363371E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052091796918016745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795712); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052091796918016745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 15:53:20,177 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 425 ms to try 5000 different iterations.  Found a solution that moves 290 regions; Going from a computed imbalance of 8.839284486363371E-4 to a new imbalance of 2.8570980496997283E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:20,178 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1662188701:82 , srv1852245428:82 , srv1084126698:83 , srv1663923836:83 , srv1876774894:83 , srv1918977088:83 , srv100843619:84 , srv1044487562:84 , srv1065818905:84 , srv1085396864:84 , srv1093820189:84 , srv1137842997:84 , srv1171399308:84 , srv1307965577:84 , srv1449132190:84 , srv1455653067:84 , srv1540580881:84 , srv1561150847:84 , srv1590007611:84 , srv1615138127:84 , srv1626704118:84 , srv1652376265:84 , srv1689771928:84 , srv1778177653:84 , srv18912068:84 , srv1918613391:84 , srv1934333882:84 , srv2012981093:84 , srv201567571:84 , srv2031012284:84 , srv2106334971:84 , srv349512453:84 , srv465723847:84 , srv502480498:84 , srv529576669:84 , srv576378557:84 , srv639445801:84 , srv647302983:84 , srv66106739:84 , srv683232945:84 , srv699089043:84 , srv789772336:84 , srv792964562:84 , srv837699534:84 , srv844640991:84 , srv968823177:84 , srv976110566:84 , srv991057595:84 , srv993520350:84 , srv999487865:84 , srv1027407816:85 , srv1116220391:85 , srv1145338614:85 , srv1149388564:85 , srv1171832959:85 , srv1214920855:85 , srv1265840734:85 , srv1275927246:85 , srv1326326719:85 , srv1357284277:85 , srv1362887715:85 , srv1364624467:85 , srv1415280477:85 , srv1442929482:85 , srv1446237579:85 , srv1452234545:85 , srv1501771116:85 , srv1523496837:85 , srv1566457530:85 , srv156805258:85 , srv1577495072:85 , srv1592811339:85 , srv1637997828:85 , srv1721019398:85 , srv1776449538:85 , srv1796264790:85 , srv1838702572:85 , srv183932182:85 , srv1879181601:85 , srv1926782625:85 , srv1940332414:85 , srv196231254:85 , srv1981222682:85 , srv2054252687:85 , srv2104159603:85 , srv21542919:85 , srv301250672:85 , srv363017103:85 , srv364430792:85 , srv440020476:85 , srv454448893:85 , srv548699470:85 , srv679820235:85 , srv737933936:85 , srv1337860894:86 , srv1472052947:86 , srv1734921054:86 , srv1851850379:86 , srv349343782:86 , srv1279654983:638 }
2024-01-03 15:53:20,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:20,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:20,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:20,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:20,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:20,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:20,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:20,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:20,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:20,305 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:20,306 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-03 15:53:20,306 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6540357095613055E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-03 15:53:20,523 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 317 ms to try 5000 different iterations.  Found a solution that moves 26 regions; Going from a computed imbalance of 2.6540357095613055E-5 to a new imbalance of 7.569813391471145E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:20,524 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1214920855:83 , srv1337860894:83 , srv1446237579:83 , srv1472052947:83 , srv1734921054:83 , srv1796264790:83 , srv1851850379:83 , srv196231254:83 , srv349343782:83 , srv349512453:83 , srv440020476:83 , srv454448893:83 , srv100843619:84 , srv1027407816:84 , srv1044487562:84 , srv1065818905:84 , srv1085396864:84 , srv1093820189:84 , srv1116220391:84 , srv1145338614:84 , srv1149388564:84 , srv1171399308:84 , srv1171832959:84 , srv1265840734:84 , srv1275927246:84 , srv1307965577:84 , srv1326326719:84 , srv1357284277:84 , srv1362887715:84 , srv1364624467:84 , srv1415280477:84 , srv1442929482:84 , srv1449132190:84 , srv1452234545:84 , srv1501771116:84 , srv1523496837:84 , srv1561150847:84 , srv1566457530:84 , srv156805258:84 , srv1577495072:84 , srv1590007611:84 , srv1592811339:84 , srv1615138127:84 , srv1637997828:84 , srv1652376265:84 , srv1662188701:84 , srv1689771928:84 , srv1721019398:84 , srv1776449538:84 , srv1778177653:84 , srv1838702572:84 , srv183932182:84 , srv1852245428:84 , srv1879181601:84 , srv18912068:84 , srv1918613391:84 , srv1926782625:84 , srv1934333882:84 , srv1940332414:84 , srv1981222682:84 , srv2012981093:84 , srv201567571:84 , srv2031012284:84 , srv2054252687:84 , srv2104159603:84 , srv2106334971:84 , srv21542919:84 , srv301250672:84 , srv363017103:84 , srv364430792:84 , srv465723847:84 , srv502480498:84 , srv529576669:84 , srv548699470:84 , srv576378557:84 , srv639445801:84 , srv647302983:84 , srv66106739:84 , srv679820235:84 , srv683232945:84 , srv699089043:84 , srv737933936:84 , srv789772336:84 , srv837699534:84 , srv844640991:84 , srv968823177:84 , srv991057595:84 , srv993520350:84 , srv1084126698:85 , srv1137842997:85 , srv1455653067:85 , srv1540580881:85 , srv1626704118:85 , srv1876774894:85 , srv1918977088:85 , srv792964562:85 , srv976110566:85 , srv999487865:85 , srv1663923836:86 , srv1279654983:684 }
2024-01-03 15:53:20,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:20,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1093820189=6, srv1265840734=14, srv1116220391=7, srv1449132190=26, srv2104159603=69, srv1084126698=4, srv789772336=91, srv1275927246=15, srv1501771116=30, srv968823177=95, srv1472052947=29, srv201567571=66, srv1663923836=44, srv647302983=85, srv1171832959=12, srv1876774894=55, srv1307965577=17, srv999487865=99, srv1027407816=1, srv1415280477=23, srv529576669=81, srv1776449538=48, srv1851850379=53, srv1357284277=20, srv1592811339=38, srv196231254=63, srv1137842997=8, srv1981222682=64, srv349343782=73, srv2031012284=67, srv548699470=82, srv301250672=72, srv576378557=83, srv1934333882=61, srv1561150847=33, srv1362887715=21, srv844640991=94, srv737933936=90, srv1326326719=18, srv66106739=86, srv1626704118=40, srv837699534=93, srv454448893=78, srv1442929482=24, srv1721019398=46, srv699089043=89, srv1637997828=41, srv440020476=77, srv1734921054=47, srv1918977088=59, srv976110566=96, srv349512453=74, srv1446237579=25, srv1838702572=51, srv1652376265=42, srv502480498=80, srv1615138127=39, srv183932182=52, srv1085396864=5, srv1879181601=56, srv1452234545=27, srv1577495072=36, srv100843619=0, srv2054252687=68, srv1279654983=16, srv1940332414=62, srv679820235=87, srv21542919=71, srv792964562=92, srv993520350=98, srv1926782625=60, srv1689771928=45, srv1214920855=13, srv1566457530=34, srv1149388564=10, srv364430792=76, srv1364624467=22, srv1778177653=49, srv363017103=75, srv1044487562=2, srv639445801=84, srv1337860894=19, srv1540580881=32, srv1662188701=43, srv465723847=79, srv1065818905=3, srv1455653067=28, srv156805258=35, srv1590007611=37, srv1796264790=50, srv1852245428=54, srv991057595=97, srv683232945=88, srv1523496837=31, srv1918613391=58, srv2106334971=70, srv18912068=57, srv1145338614=9, srv1171399308=11, srv2012981093=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:20,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-03 15:53:20,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-03 15:53:20,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-03 15:53:20,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-03 15:53:20,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-03 15:53:20,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-03 15:53:20,658 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:20,658 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1214920855:83 , srv1337860894:83 , srv1446237579:83 , srv1472052947:83 , srv1734921054:83 , srv1796264790:83 , srv1851850379:83 , srv196231254:83 , srv349343782:83 , srv349512453:83 , srv440020476:83 , srv454448893:83 , srv100843619:84 , srv1027407816:84 , srv1044487562:84 , srv1065818905:84 , srv1085396864:84 , srv1093820189:84 , srv1116220391:84 , srv1145338614:84 , srv1149388564:84 , srv1171399308:84 , srv1171832959:84 , srv1265840734:84 , srv1275927246:84 , srv1307965577:84 , srv1326326719:84 , srv1357284277:84 , srv1362887715:84 , srv1364624467:84 , srv1415280477:84 , srv1442929482:84 , srv1449132190:84 , srv1452234545:84 , srv1501771116:84 , srv1523496837:84 , srv1561150847:84 , srv1566457530:84 , srv156805258:84 , srv1577495072:84 , srv1590007611:84 , srv1592811339:84 , srv1615138127:84 , srv1637997828:84 , srv1652376265:84 , srv1662188701:84 , srv1689771928:84 , srv1721019398:84 , srv1776449538:84 , srv1778177653:84 , srv1838702572:84 , srv183932182:84 , srv1852245428:84 , srv1879181601:84 , srv18912068:84 , srv1918613391:84 , srv1926782625:84 , srv1934333882:84 , srv1940332414:84 , srv1981222682:84 , srv2012981093:84 , srv201567571:84 , srv2031012284:84 , srv2054252687:84 , srv2104159603:84 , srv2106334971:84 , srv21542919:84 , srv301250672:84 , srv363017103:84 , srv364430792:84 , srv465723847:84 , srv502480498:84 , srv529576669:84 , srv548699470:84 , srv576378557:84 , srv639445801:84 , srv647302983:84 , srv66106739:84 , srv679820235:84 , srv683232945:84 , srv699089043:84 , srv737933936:84 , srv789772336:84 , srv837699534:84 , srv844640991:84 , srv968823177:84 , srv991057595:84 , srv993520350:84 , srv1084126698:85 , srv1137842997:85 , srv1455653067:85 , srv1540580881:85 , srv1626704118:85 , srv1876774894:85 , srv1918977088:85 , srv792964562:85 , srv976110566:85 , srv999487865:85 , srv1663923836:86 , srv1279654983:684 }
2024-01-03 15:53:20,689 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 298), ProcessCount=167 (was 167), AvailableMemoryMB=3879 (was 4464)
2024-01-03 15:53:20,697 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=290, ProcessCount=167, AvailableMemoryMB=3879
2024-01-03 15:53:20,697 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-03 15:53:20,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-03 15:53:20,707 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv105976242:84 , srv1204473960:84 , srv1208498789:84 , srv1443364885:84 , srv1457732436:84 , srv1487522814:84 , srv1511997669:84 , srv1607957601:84 , srv163776753:84 , srv1676952175:84 , srv1860788041:84 , srv1929798223:84 , srv1971128248:84 , srv199797535:84 , srv276201485:84 , srv292610980:84 , srv392790198:84 , srv443082859:84 , srv511181884:84 , srv518210066:84 , srv532898794:84 , srv677750088:84 , srv723841987:84 , srv72414352:84 , srv73399769:84 , srv789865960:84 , srv82548678:84 , srv841122972:84 , srv885241515:84 , srv514317313:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-03 15:53:20,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:53:20,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1929798223=11, srv885241515=29, srv276201485=14, srv514317313=19, srv1487522814=5, srv1607957601=7, srv199797535=13, srv723841987=23, srv1971128248=12, srv1457732436=4, srv532898794=21, srv292610980=15, srv511181884=18, srv1204473960=1, srv841122972=28, srv443082859=17, srv1676952175=9, srv789865960=26, srv73399769=25, srv163776753=8, srv392790198=16, srv105976242=0, srv72414352=24, srv82548678=27, srv1860788041=10, srv1443364885=3, srv1208498789=2, srv1511997669=6, srv677750088=22, srv518210066=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:53:20,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:53:20,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:53:20,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:53:20,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:53:20,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:53:20,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:53:20,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:53:20,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:53:20,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:53:20,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-03 15:53:20,733 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1770972407815523, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:53:20,733 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912751070887225, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1770972407815523, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-03 15:55:20,721 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6197828 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912751070887225 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:55:20,727 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105976242:90 , srv1204473960:90 , srv1208498789:90 , srv1443364885:90 , srv1457732436:90 , srv1487522814:90 , srv1511997669:90 , srv1607957601:90 , srv163776753:90 , srv1676952175:90 , srv1860788041:90 , srv1929798223:90 , srv1971128248:90 , srv199797535:90 , srv276201485:90 , srv292610980:90 , srv392790198:90 , srv443082859:90 , srv511181884:90 , srv514317313:90 , srv518210066:90 , srv532898794:90 , srv677750088:90 , srv723841987:90 , srv72414352:90 , srv73399769:90 , srv789865960:90 , srv82548678:90 , srv841122972:90 , srv885241515:90 }
2024-01-03 15:55:20,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-03 15:55:20,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1929798223=11, srv885241515=29, srv276201485=14, srv514317313=19, srv1487522814=5, srv1607957601=7, srv199797535=13, srv723841987=23, srv1971128248=12, srv1457732436=4, srv532898794=21, srv292610980=15, srv511181884=18, srv1204473960=1, srv841122972=28, srv443082859=17, srv1676952175=9, srv789865960=26, srv73399769=25, srv163776753=8, srv392790198=16, srv105976242=0, srv72414352=24, srv82548678=27, srv1860788041=10, srv1443364885=3, srv1208498789=2, srv1511997669=6, srv677750088=22, srv518210066=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-03 15:55:20,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-03 15:55:20,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-03 15:55:20,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-03 15:55:20,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-03 15:55:20,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-03 15:55:20,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-03 15:55:20,746 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-03 15:55:20,746 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv105976242:90 , srv1204473960:90 , srv1208498789:90 , srv1443364885:90 , srv1457732436:90 , srv1487522814:90 , srv1511997669:90 , srv1607957601:90 , srv163776753:90 , srv1676952175:90 , srv1860788041:90 , srv1929798223:90 , srv1971128248:90 , srv199797535:90 , srv276201485:90 , srv292610980:90 , srv392790198:90 , srv443082859:90 , srv511181884:90 , srv514317313:90 , srv518210066:90 , srv532898794:90 , srv677750088:90 , srv723841987:90 , srv72414352:90 , srv73399769:90 , srv789865960:90 , srv82548678:90 , srv841122972:90 , srv885241515:90 }
2024-01-03 15:55:20,763 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 290), ProcessCount=167 (was 167), AvailableMemoryMB=3370 (was 3879)