2023-04-30 12:53:35,434 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-30 12:53:35,762 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-30 12:53:35,893 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-30 12:53:35,953 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-30 12:53:35,961 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-30 12:53:35,974 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=170, AvailableMemoryMB=11250
2023-04-30 12:53:35,976 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-30 12:53:35,977 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-30 12:53:36,116 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008764735:84 , srv1024521304:84 , srv1044027945:84 , srv1077347848:84 , srv1125627330:84 , srv1126437548:84 , srv1173276013:84 , srv1221246651:84 , srv1226175081:84 , srv1289161042:84 , srv1312308119:84 , srv1313217033:84 , srv1328935801:84 , srv132917924:84 , srv1356452888:84 , srv1431388693:84 , srv1441033954:84 , srv1486803127:84 , srv1488139143:84 , srv1505855:84 , srv1509715608:84 , srv1541830982:84 , srv1559427911:84 , srv1575736942:84 , srv1584516953:84 , srv1597611866:84 , srv1683720499:84 , srv1690869054:84 , srv1702439936:84 , srv1734273257:84 , srv1734769350:84 , srv1769448719:84 , srv178629580:84 , srv181244189:84 , srv1812616131:84 , srv1825221435:84 , srv1849712050:84 , srv1854381234:84 , srv1860424695:84 , srv1893234459:84 , srv1896382382:84 , srv1909678117:84 , srv1970340015:84 , srv1984255450:84 , srv1994191854:84 , srv1998021057:84 , srv1998079129:84 , srv1999773952:84 , srv202847316:84 , srv2081387883:84 , srv2112115061:84 , srv2112213864:84 , srv2135951220:84 , srv223313402:84 , srv225713781:84 , srv250270852:84 , srv264357255:84 , srv279011131:84 , srv292893552:84 , srv35320352:84 , srv353656347:84 , srv360749894:84 , srv364079119:84 , srv381982235:84 , srv411148950:84 , srv413447261:84 , srv423976549:84 , srv461426274:84 , srv465473600:84 , srv520825625:84 , srv548653339:84 , srv569493223:84 , srv576792788:84 , srv578632088:84 , srv584773441:84 , srv628560556:84 , srv643683678:84 , srv649629168:84 , srv684577861:84 , srv720538901:84 , srv721769022:84 , srv735530285:84 , srv749663896:84 , srv76773447:84 , srv790860302:84 , srv793285352:84 , srv797878826:84 , srv801490477:84 , srv839684204:84 , srv872232650:84 , srv885843498:84 , srv914001040:84 , srv922140421:84 , srv947094060:84 , srv954286001:84 , srv955131647:84 , srv964017671:84 , srv983851756:84 , srv993308486:84 , srv152473338:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-30 12:53:36,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:36,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1994191854=45, srv202847316=49, srv793285352=86, srv2135951220=53, srv735530285=82, srv993308486=99, srv1998079129=47, srv576792788=73, srv411148950=65, srv1854381234=38, srv413447261=66, srv1289161042=9, srv1734769350=31, srv839684204=89, srv461426274=68, srv1008764735=0, srv790860302=85, srv1077347848=3, srv1683720499=27, srv465473600=69, srv1734273257=30, srv584773441=75, srv1044027945=2, srv2081387883=50, srv947094060=94, srv1812616131=35, srv721769022=81, srv520825625=70, srv578632088=74, srv1584516953=25, srv548653339=71, srv1970340015=43, srv279011131=58, srv364079119=63, srv1825221435=36, srv872232650=90, srv1509715608=20, srv1849712050=37, srv1769448719=32, srv178629580=33, srv264357255=57, srv1125627330=4, srv1173276013=6, srv964017671=97, srv1488139143=18, srv225713781=55, srv381982235=64, srv922140421=93, srv2112213864=52, srv643683678=77, srv885843498=91, srv1893234459=40, srv1575736942=24, srv1597611866=26, srv1226175081=8, srv1486803127=17, srv1541830982=22, srv132917924=13, srv1328935801=12, srv720538901=80, srv749663896=83, srv1312308119=10, srv1984255450=44, srv649629168=78, srv1024521304=1, srv1702439936=29, srv1313217033=11, srv223313402=54, srv1860424695=39, srv954286001=95, srv684577861=79, srv1909678117=42, srv628560556=76, srv1998021057=46, srv360749894=62, srv292893552=59, srv152473338=21, srv1505855=19, srv353656347=61, srv2112115061=51, srv1126437548=5, srv1896382382=41, srv955131647=96, srv983851756=98, srv1431388693=15, srv1221246651=7, srv914001040=92, srv181244189=34, srv423976549=67, srv569493223=72, srv1441033954=16, srv1559427911=23, srv1999773952=48, srv1690869054=28, srv250270852=56, srv35320352=60, srv76773447=84, srv797878826=87, srv801490477=88, srv1356452888=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:36,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:36,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:36,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-30 12:53:36,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-30 12:53:36,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-30 12:53:36,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-30 12:53:36,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-30 12:53:36,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-30 12:53:36,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-30 12:53:36,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-30 12:53:36,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-30 12:53:36,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-30 12:53:36,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-30 12:53:36,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-30 12:53:36,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-30 12:53:36,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-30 12:53:36,698 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:36,699 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-30 12:53:36,699 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-30 12:53:37,440 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1153 ms to try 5000 different iterations.  Found a solution that moves 4733 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17751634791042625. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04252651879824392, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05976760080093092, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5258888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04252651879824392, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17241666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:37,451 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2112115061:81 , srv983851756:81 , srv1077347848:82 , srv1312308119:82 , srv1702439936:82 , srv353656347:82 , srv413447261:82 , srv1008764735:83 , srv1541830982:83 , srv1734273257:83 , srv423976549:83 , srv520825625:83 , srv793285352:83 , srv922140421:83 , srv947094060:83 , srv1356452888:84 , srv1488139143:84 , srv1505855:84 , srv1509715608:84 , srv1597611866:84 , srv1683720499:84 , srv1893234459:84 , srv1998079129:84 , srv2081387883:84 , srv279011131:84 , srv360749894:84 , srv569493223:84 , srv643683678:84 , srv649629168:84 , srv720538901:84 , srv790860302:84 , srv801490477:84 , srv914001040:84 , srv954286001:84 , srv1126437548:85 , srv1313217033:85 , srv1431388693:85 , srv1441033954:85 , srv1584516953:85 , srv1854381234:85 , srv1860424695:85 , srv578632088:85 , srv684577861:85 , srv735530285:85 , srv1289161042:86 , srv225713781:86 , srv465473600:86 , srv628560556:86 , srv839684204:86 , srv1044027945:87 , srv1125627330:87 , srv1173276013:87 , srv132917924:87 , srv1486803127:87 , srv1575736942:87 , srv1734769350:87 , srv181244189:87 , srv2112213864:87 , srv2135951220:87 , srv223313402:87 , srv35320352:87 , srv411148950:87 , srv548653339:87 , srv576792788:87 , srv885843498:87 , srv1024521304:88 , srv1226175081:88 , srv1328935801:88 , srv1559427911:88 , srv178629580:88 , srv1812616131:88 , srv1909678117:88 , srv1970340015:88 , srv1984255450:88 , srv1999773952:88 , srv364079119:88 , srv381982235:88 , srv76773447:88 , srv797878826:88 , srv872232650:88 , srv955131647:88 , srv1221246651:89 , srv1849712050:89 , srv1994191854:89 , srv1998021057:89 , srv264357255:89 , srv292893552:89 , srv721769022:89 , srv749663896:89 , srv964017671:89 , srv993308486:89 , srv1690869054:90 , srv1825221435:90 , srv202847316:90 , srv584773441:90 , srv461426274:91 , srv1769448719:92 , srv1896382382:92 , srv250270852:95 , srv152473338:468 }
2023-04-30 12:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1994191854=45, srv202847316=49, srv793285352=86, srv2135951220=53, srv735530285=82, srv993308486=99, srv1998079129=47, srv576792788=73, srv411148950=65, srv1854381234=38, srv413447261=66, srv1289161042=9, srv1734769350=31, srv839684204=89, srv461426274=68, srv1008764735=0, srv790860302=85, srv1077347848=3, srv1683720499=27, srv465473600=69, srv1734273257=30, srv584773441=75, srv1044027945=2, srv2081387883=50, srv947094060=94, srv1812616131=35, srv721769022=81, srv520825625=70, srv578632088=74, srv1584516953=25, srv548653339=71, srv1970340015=43, srv279011131=58, srv364079119=63, srv1825221435=36, srv872232650=90, srv1509715608=20, srv1849712050=37, srv1769448719=32, srv178629580=33, srv264357255=57, srv1125627330=4, srv1173276013=6, srv964017671=97, srv1488139143=18, srv225713781=55, srv381982235=64, srv922140421=93, srv2112213864=52, srv643683678=77, srv885843498=91, srv1893234459=40, srv1575736942=24, srv1597611866=26, srv1226175081=8, srv1486803127=17, srv1541830982=22, srv132917924=13, srv1328935801=12, srv720538901=80, srv749663896=83, srv1312308119=10, srv1984255450=44, srv649629168=78, srv1024521304=1, srv1702439936=29, srv1313217033=11, srv223313402=54, srv1860424695=39, srv954286001=95, srv684577861=79, srv1909678117=42, srv628560556=76, srv1998021057=46, srv360749894=62, srv292893552=59, srv152473338=21, srv1505855=19, srv353656347=61, srv2112115061=51, srv1126437548=5, srv1896382382=41, srv955131647=96, srv983851756=98, srv1431388693=15, srv1221246651=7, srv914001040=92, srv181244189=34, srv423976549=67, srv569493223=72, srv1441033954=16, srv1559427911=23, srv1999773952=48, srv1690869054=28, srv250270852=56, srv35320352=60, srv76773447=84, srv797878826=87, srv801490477=88, srv1356452888=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:37,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:37,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:37,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:37,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:37,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:37,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-30 12:53:37,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-30 12:53:37,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-30 12:53:37,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-30 12:53:37,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-30 12:53:37,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-30 12:53:37,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-30 12:53:37,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-30 12:53:37,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-30 12:53:37,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-30 12:53:37,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-30 12:53:37,666 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04252651879824392, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05976760080093092, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04252651879824392, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17241666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:37,666 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-30 12:53:37,667 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17748320673608572, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04252651879824392, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05976760080093092, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04252651879824392, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17241666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-30 12:53:37,989 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 464 ms to try 5000 different iterations.  Found a solution that moves 2600 regions; Going from a computed imbalance of 0.17748320673608572 to a new imbalance of 0.015080542540413336. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02394812950621071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03886723784022791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02394812950621071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:38,000 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv250270852:76 , srv1690869054:81 , srv1896382382:81 , srv202847316:81 , srv1221246651:82 , srv1769448719:82 , srv178629580:82 , srv292893552:82 , srv584773441:82 , srv721769022:82 , srv749663896:82 , srv964017671:82 , srv993308486:82 , srv1486803127:83 , srv1559427911:83 , srv1575736942:83 , srv1812616131:83 , srv1825221435:83 , srv1849712050:83 , srv1994191854:83 , srv1998021057:83 , srv1999773952:83 , srv264357255:83 , srv364079119:83 , srv411148950:83 , srv955131647:83 , srv1024521304:84 , srv1044027945:84 , srv1734769350:84 , srv1909678117:84 , srv1970340015:84 , srv2112213864:84 , srv461426274:84 , srv76773447:84 , srv839684204:84 , srv872232650:84 , srv885843498:84 , srv1125627330:85 , srv1126437548:85 , srv1226175081:85 , srv1289161042:85 , srv1328935801:85 , srv1984255450:85 , srv223313402:85 , srv35320352:85 , srv381982235:85 , srv548653339:85 , srv132917924:86 , srv1441033954:86 , srv1584516953:86 , srv1854381234:86 , srv2135951220:86 , srv225713781:86 , srv465473600:86 , srv576792788:86 , srv628560556:86 , srv720538901:86 , srv735530285:86 , srv1173276013:87 , srv1313217033:87 , srv1431388693:87 , srv1509715608:87 , srv1683720499:87 , srv1734273257:87 , srv1998079129:87 , srv2081387883:87 , srv578632088:87 , srv790860302:87 , srv797878826:87 , srv922140421:87 , srv1008764735:88 , srv1356452888:88 , srv1505855:88 , srv181244189:88 , srv1860424695:88 , srv279011131:88 , srv353656347:88 , srv520825625:88 , srv569493223:88 , srv649629168:88 , srv793285352:88 , srv801490477:88 , srv914001040:88 , srv954286001:88 , srv1312308119:89 , srv1541830982:89 , srv1597611866:89 , srv1702439936:89 , srv1893234459:89 , srv2112115061:89 , srv413447261:89 , srv684577861:89 , srv1077347848:90 , srv423976549:90 , srv643683678:90 , srv1488139143:91 , srv360749894:91 , srv947094060:91 , srv983851756:94 , srv152473338:519 }
2023-04-30 12:53:38,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:38,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1994191854=45, srv202847316=49, srv793285352=86, srv2135951220=53, srv735530285=82, srv993308486=99, srv1998079129=47, srv576792788=73, srv411148950=65, srv1854381234=38, srv413447261=66, srv1289161042=9, srv1734769350=31, srv839684204=89, srv461426274=68, srv1008764735=0, srv790860302=85, srv1077347848=3, srv1683720499=27, srv465473600=69, srv1734273257=30, srv584773441=75, srv1044027945=2, srv2081387883=50, srv947094060=94, srv1812616131=35, srv721769022=81, srv520825625=70, srv578632088=74, srv1584516953=25, srv548653339=71, srv1970340015=43, srv279011131=58, srv364079119=63, srv1825221435=36, srv872232650=90, srv1509715608=20, srv1849712050=37, srv1769448719=32, srv178629580=33, srv264357255=57, srv1125627330=4, srv1173276013=6, srv964017671=97, srv1488139143=18, srv225713781=55, srv381982235=64, srv922140421=93, srv2112213864=52, srv643683678=77, srv885843498=91, srv1893234459=40, srv1575736942=24, srv1597611866=26, srv1226175081=8, srv1486803127=17, srv1541830982=22, srv132917924=13, srv1328935801=12, srv720538901=80, srv749663896=83, srv1312308119=10, srv1984255450=44, srv649629168=78, srv1024521304=1, srv1702439936=29, srv1313217033=11, srv223313402=54, srv1860424695=39, srv954286001=95, srv684577861=79, srv1909678117=42, srv628560556=76, srv1998021057=46, srv360749894=62, srv292893552=59, srv152473338=21, srv1505855=19, srv353656347=61, srv2112115061=51, srv1126437548=5, srv1896382382=41, srv955131647=96, srv983851756=98, srv1431388693=15, srv1221246651=7, srv914001040=92, srv181244189=34, srv423976549=67, srv569493223=72, srv1441033954=16, srv1559427911=23, srv1999773952=48, srv1690869054=28, srv250270852=56, srv35320352=60, srv76773447=84, srv797878826=87, srv801490477=88, srv1356452888=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:38,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:38,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-30 12:53:38,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-30 12:53:38,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-30 12:53:38,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-30 12:53:38,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-30 12:53:38,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-30 12:53:38,136 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02394812950621071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03886723784022791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02394812950621071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:38,136 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-30 12:53:38,136 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015062336951297477, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02394812950621071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03886723784022791, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02394812950621071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-30 12:53:38,503 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations.  Found a solution that moves 974 regions; Going from a computed imbalance of 0.015062336951297477 to a new imbalance of 7.822053524700712E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007780182408466789); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01226347491779894); MoveCostFunction : (multiplier=7.0, imbalance=0.10822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007780182408466789); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:38,504 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1488139143:82 , srv181244189:82 , srv983851756:82 , srv1173276013:83 , srv1825221435:83 , srv461426274:83 , srv643683678:83 , srv684577861:83 , srv797878826:83 , srv947094060:83 , srv1226175081:84 , srv1313217033:84 , srv1328935801:84 , srv1505855:84 , srv1541830982:84 , srv1769448719:84 , srv1860424695:84 , srv1893234459:84 , srv1909678117:84 , srv360749894:84 , srv381982235:84 , srv423976549:84 , srv578632088:84 , srv1024521304:85 , srv1077347848:85 , srv1312308119:85 , srv132917924:85 , srv1356452888:85 , srv1597611866:85 , srv1849712050:85 , srv1854381234:85 , srv1896382382:85 , srv1984255450:85 , srv1994191854:85 , srv1998021057:85 , srv2135951220:85 , srv223313402:85 , srv279011131:85 , srv292893552:85 , srv35320352:85 , srv569493223:85 , srv576792788:85 , srv628560556:85 , srv76773447:85 , srv793285352:85 , srv801490477:85 , srv872232650:85 , srv1125627330:86 , srv1289161042:86 , srv1431388693:86 , srv1584516953:86 , srv1683720499:86 , srv1702439936:86 , srv1812616131:86 , srv1970340015:86 , srv202847316:86 , srv2112213864:86 , srv225713781:86 , srv250270852:86 , srv264357255:86 , srv413447261:86 , srv465473600:86 , srv520825625:86 , srv548653339:86 , srv584773441:86 , srv649629168:86 , srv721769022:86 , srv735530285:86 , srv749663896:86 , srv790860302:86 , srv885843498:86 , srv914001040:86 , srv964017671:86 , srv1008764735:87 , srv1044027945:87 , srv1126437548:87 , srv1221246651:87 , srv1441033954:87 , srv1486803127:87 , srv1509715608:87 , srv1559427911:87 , srv1575736942:87 , srv1690869054:87 , srv1734769350:87 , srv1998079129:87 , srv1999773952:87 , srv2081387883:87 , srv2112115061:87 , srv353656347:87 , srv364079119:87 , srv411148950:87 , srv839684204:87 , srv922140421:87 , srv954286001:87 , srv955131647:87 , srv993308486:87 , srv1734273257:88 , srv178629580:88 , srv720538901:88 , srv152473338:540 }
2023-04-30 12:53:38,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:38,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1994191854=45, srv202847316=49, srv793285352=86, srv2135951220=53, srv735530285=82, srv993308486=99, srv1998079129=47, srv576792788=73, srv411148950=65, srv1854381234=38, srv413447261=66, srv1289161042=9, srv1734769350=31, srv839684204=89, srv461426274=68, srv1008764735=0, srv790860302=85, srv1077347848=3, srv1683720499=27, srv465473600=69, srv1734273257=30, srv584773441=75, srv1044027945=2, srv2081387883=50, srv947094060=94, srv1812616131=35, srv721769022=81, srv520825625=70, srv578632088=74, srv1584516953=25, srv548653339=71, srv1970340015=43, srv279011131=58, srv364079119=63, srv1825221435=36, srv872232650=90, srv1509715608=20, srv1849712050=37, srv1769448719=32, srv178629580=33, srv264357255=57, srv1125627330=4, srv1173276013=6, srv964017671=97, srv1488139143=18, srv225713781=55, srv381982235=64, srv922140421=93, srv2112213864=52, srv643683678=77, srv885843498=91, srv1893234459=40, srv1575736942=24, srv1597611866=26, srv1226175081=8, srv1486803127=17, srv1541830982=22, srv132917924=13, srv1328935801=12, srv720538901=80, srv749663896=83, srv1312308119=10, srv1984255450=44, srv649629168=78, srv1024521304=1, srv1702439936=29, srv1313217033=11, srv223313402=54, srv1860424695=39, srv954286001=95, srv684577861=79, srv1909678117=42, srv628560556=76, srv1998021057=46, srv360749894=62, srv292893552=59, srv152473338=21, srv1505855=19, srv353656347=61, srv2112115061=51, srv1126437548=5, srv1896382382=41, srv955131647=96, srv983851756=98, srv1431388693=15, srv1221246651=7, srv914001040=92, srv181244189=34, srv423976549=67, srv569493223=72, srv1441033954=16, srv1559427911=23, srv1999773952=48, srv1690869054=28, srv250270852=56, srv35320352=60, srv76773447=84, srv797878826=87, srv801490477=88, srv1356452888=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:38,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:38,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:38,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:38,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:38,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:38,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:38,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-30 12:53:38,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-30 12:53:38,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-30 12:53:38,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-30 12:53:38,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-30 12:53:38,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-30 12:53:38,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-30 12:53:38,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-30 12:53:38,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-30 12:53:38,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-30 12:53:38,672 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007780182408466789); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01226347491779894); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007780182408466789); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:38,673 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-30 12:53:38,673 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.753852587012841E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007780182408466789); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01226347491779894); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007780182408466789); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-30 12:53:38,953 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 389 ms to try 5000 different iterations.  Found a solution that moves 275 regions; Going from a computed imbalance of 7.753852587012841E-4 to a new imbalance of 2.0998801155519673E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:38,954 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv132917924:83 , srv1597611866:83 , srv360749894:83 , srv465473600:83 , srv576792788:83 , srv954286001:83 , srv1008764735:84 , srv1044027945:84 , srv1125627330:84 , srv1126437548:84 , srv1221246651:84 , srv1431388693:84 , srv1441033954:84 , srv1486803127:84 , srv1509715608:84 , srv1559427911:84 , srv1702439936:84 , srv1734273257:84 , srv1734769350:84 , srv1769448719:84 , srv178629580:84 , srv1849712050:84 , srv1896382382:84 , srv1970340015:84 , srv1984255450:84 , srv1998079129:84 , srv1999773952:84 , srv202847316:84 , srv2135951220:84 , srv225713781:84 , srv264357255:84 , srv461426274:84 , srv548653339:84 , srv584773441:84 , srv643683678:84 , srv649629168:84 , srv720538901:84 , srv955131647:84 , srv993308486:84 , srv1024521304:85 , srv1077347848:85 , srv1173276013:85 , srv1226175081:85 , srv1289161042:85 , srv1312308119:85 , srv1328935801:85 , srv1356452888:85 , srv1488139143:85 , srv1575736942:85 , srv1584516953:85 , srv1683720499:85 , srv1690869054:85 , srv181244189:85 , srv1860424695:85 , srv1893234459:85 , srv1994191854:85 , srv1998021057:85 , srv2081387883:85 , srv2112213864:85 , srv223313402:85 , srv250270852:85 , srv279011131:85 , srv353656347:85 , srv364079119:85 , srv381982235:85 , srv411148950:85 , srv413447261:85 , srv423976549:85 , srv520825625:85 , srv569493223:85 , srv628560556:85 , srv684577861:85 , srv721769022:85 , srv735530285:85 , srv76773447:85 , srv790860302:85 , srv797878826:85 , srv839684204:85 , srv872232650:85 , srv885843498:85 , srv914001040:85 , srv922140421:85 , srv947094060:85 , srv964017671:85 , srv983851756:85 , srv1313217033:86 , srv1505855:86 , srv1541830982:86 , srv1812616131:86 , srv1825221435:86 , srv1854381234:86 , srv2112115061:86 , srv292893552:86 , srv35320352:86 , srv578632088:86 , srv749663896:86 , srv793285352:86 , srv801490477:86 , srv1909678117:87 , srv152473338:615 }
2023-04-30 12:53:38,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:38,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1994191854=45, srv202847316=49, srv793285352=86, srv2135951220=53, srv735530285=82, srv993308486=99, srv1998079129=47, srv576792788=73, srv411148950=65, srv1854381234=38, srv413447261=66, srv1289161042=9, srv1734769350=31, srv839684204=89, srv461426274=68, srv1008764735=0, srv790860302=85, srv1077347848=3, srv1683720499=27, srv465473600=69, srv1734273257=30, srv584773441=75, srv1044027945=2, srv2081387883=50, srv947094060=94, srv1812616131=35, srv721769022=81, srv520825625=70, srv578632088=74, srv1584516953=25, srv548653339=71, srv1970340015=43, srv279011131=58, srv364079119=63, srv1825221435=36, srv872232650=90, srv1509715608=20, srv1849712050=37, srv1769448719=32, srv178629580=33, srv264357255=57, srv1125627330=4, srv1173276013=6, srv964017671=97, srv1488139143=18, srv225713781=55, srv381982235=64, srv922140421=93, srv2112213864=52, srv643683678=77, srv885843498=91, srv1893234459=40, srv1575736942=24, srv1597611866=26, srv1226175081=8, srv1486803127=17, srv1541830982=22, srv132917924=13, srv1328935801=12, srv720538901=80, srv749663896=83, srv1312308119=10, srv1984255450=44, srv649629168=78, srv1024521304=1, srv1702439936=29, srv1313217033=11, srv223313402=54, srv1860424695=39, srv954286001=95, srv684577861=79, srv1909678117=42, srv628560556=76, srv1998021057=46, srv360749894=62, srv292893552=59, srv152473338=21, srv1505855=19, srv353656347=61, srv2112115061=51, srv1126437548=5, srv1896382382=41, srv955131647=96, srv983851756=98, srv1431388693=15, srv1221246651=7, srv914001040=92, srv181244189=34, srv423976549=67, srv569493223=72, srv1441033954=16, srv1559427911=23, srv1999773952=48, srv1690869054=28, srv250270852=56, srv35320352=60, srv76773447=84, srv797878826=87, srv801490477=88, srv1356452888=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:38,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:38,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-30 12:53:38,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-30 12:53:38,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-30 12:53:38,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-30 12:53:38,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-30 12:53:38,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-30 12:53:38,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-30 12:53:38,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-30 12:53:38,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-30 12:53:38,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-30 12:53:38,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-30 12:53:38,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-30 12:53:39,149 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9073209999034633E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:39,150 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv132917924:83 , srv1597611866:83 , srv360749894:83 , srv465473600:83 , srv576792788:83 , srv954286001:83 , srv1008764735:84 , srv1044027945:84 , srv1125627330:84 , srv1126437548:84 , srv1221246651:84 , srv1431388693:84 , srv1441033954:84 , srv1486803127:84 , srv1509715608:84 , srv1559427911:84 , srv1702439936:84 , srv1734273257:84 , srv1734769350:84 , srv1769448719:84 , srv178629580:84 , srv1849712050:84 , srv1896382382:84 , srv1970340015:84 , srv1984255450:84 , srv1998079129:84 , srv1999773952:84 , srv202847316:84 , srv2135951220:84 , srv225713781:84 , srv264357255:84 , srv461426274:84 , srv548653339:84 , srv584773441:84 , srv643683678:84 , srv649629168:84 , srv720538901:84 , srv955131647:84 , srv993308486:84 , srv1024521304:85 , srv1077347848:85 , srv1173276013:85 , srv1226175081:85 , srv1289161042:85 , srv1312308119:85 , srv1328935801:85 , srv1356452888:85 , srv1488139143:85 , srv1575736942:85 , srv1584516953:85 , srv1683720499:85 , srv1690869054:85 , srv181244189:85 , srv1860424695:85 , srv1893234459:85 , srv1994191854:85 , srv1998021057:85 , srv2081387883:85 , srv2112213864:85 , srv223313402:85 , srv250270852:85 , srv279011131:85 , srv353656347:85 , srv364079119:85 , srv381982235:85 , srv411148950:85 , srv413447261:85 , srv423976549:85 , srv520825625:85 , srv569493223:85 , srv628560556:85 , srv684577861:85 , srv721769022:85 , srv735530285:85 , srv76773447:85 , srv790860302:85 , srv797878826:85 , srv839684204:85 , srv872232650:85 , srv885843498:85 , srv914001040:85 , srv922140421:85 , srv947094060:85 , srv964017671:85 , srv983851756:85 , srv1313217033:86 , srv1505855:86 , srv1541830982:86 , srv1812616131:86 , srv1825221435:86 , srv1854381234:86 , srv2112115061:86 , srv292893552:86 , srv35320352:86 , srv578632088:86 , srv749663896:86 , srv793285352:86 , srv801490477:86 , srv1909678117:87 , srv152473338:615 }
2023-04-30 12:53:39,183 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=342 (was 328) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10370 (was 11250)
2023-04-30 12:53:39,191 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=170, AvailableMemoryMB=10370
2023-04-30 12:53:39,192 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-30 12:53:39,192 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-30 12:53:39,204 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009158183:84 , srv1011903959:84 , srv1130995835:84 , srv117634198:84 , srv117750757:84 , srv1330102218:84 , srv1511214126:84 , srv1513723774:84 , srv1721163913:84 , srv1750854367:84 , srv1799100124:84 , srv1801746185:84 , srv1908504314:84 , srv1927288417:84 , srv1976235117:84 , srv2037984631:84 , srv2064429993:84 , srv337761351:84 , srv34452118:84 , srv380975363:84 , srv510321662:84 , srv605161776:84 , srv741271068:84 , srv78284150:84 , srv858058090:84 , srv916702815:84 , srv935842176:84 , srv93931295:84 , srv941064293:84 , srv2064053337:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-30 12:53:39,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:53:39,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2064053337=16, srv2064429993=17, srv117750757=4, srv337761351=18, srv1130995835=2, srv2037984631=15, srv93931295=28, srv1330102218=5, srv78284150=24, srv1009158183=0, srv1908504314=12, srv1721163913=8, srv1511214126=6, srv1976235117=14, srv935842176=27, srv1513723774=7, srv34452118=19, srv1011903959=1, srv117634198=3, srv1927288417=13, srv510321662=21, srv605161776=22, srv741271068=23, srv1750854367=9, srv916702815=26, srv1801746185=11, srv858058090=25, srv941064293=29, srv1799100124=10, srv380975363=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:53:39,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:53:39,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:53:39,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:53:39,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-30 12:53:39,245 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1866722725854428, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:53:39,245 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912781241499784, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1866722725854428, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-30 12:55:39,224 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6907722 different iterations.  Found a solution that moves 1869 regions; Going from a computed imbalance of 0.9912781241499784 to a new imbalance of 4.362339238146111E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:55:39,231 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1009158183:90 , srv1011903959:90 , srv1130995835:90 , srv117634198:90 , srv117750757:90 , srv1330102218:90 , srv1511214126:90 , srv1513723774:90 , srv1721163913:90 , srv1750854367:90 , srv1799100124:90 , srv1801746185:90 , srv1908504314:90 , srv1927288417:90 , srv1976235117:90 , srv2037984631:90 , srv2064053337:90 , srv2064429993:90 , srv337761351:90 , srv34452118:90 , srv380975363:90 , srv510321662:90 , srv605161776:90 , srv741271068:90 , srv78284150:90 , srv858058090:90 , srv916702815:90 , srv935842176:90 , srv93931295:90 , srv941064293:90 }
2023-04-30 12:55:39,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-30 12:55:39,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2064053337=16, srv2064429993=17, srv117750757=4, srv337761351=18, srv1130995835=2, srv2037984631=15, srv93931295=28, srv1330102218=5, srv78284150=24, srv1009158183=0, srv1908504314=12, srv1721163913=8, srv1511214126=6, srv1976235117=14, srv935842176=27, srv1513723774=7, srv34452118=19, srv1011903959=1, srv117634198=3, srv1927288417=13, srv510321662=21, srv605161776=22, srv741271068=23, srv1750854367=9, srv916702815=26, srv1801746185=11, srv858058090=25, srv941064293=29, srv1799100124=10, srv380975363=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-30 12:55:39,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-30 12:55:39,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-30 12:55:39,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-30 12:55:39,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-30 12:55:39,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-30 12:55:39,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-30 12:55:39,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-30 12:55:39,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-30 12:55:39,255 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-30 12:55:39,255 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1009158183:90 , srv1011903959:90 , srv1130995835:90 , srv117634198:90 , srv117750757:90 , srv1330102218:90 , srv1511214126:90 , srv1513723774:90 , srv1721163913:90 , srv1750854367:90 , srv1799100124:90 , srv1801746185:90 , srv1908504314:90 , srv1927288417:90 , srv1976235117:90 , srv2037984631:90 , srv2064053337:90 , srv2064429993:90 , srv337761351:90 , srv34452118:90 , srv380975363:90 , srv510321662:90 , srv605161776:90 , srv741271068:90 , srv78284150:90 , srv858058090:90 , srv916702815:90 , srv935842176:90 , srv93931295:90 , srv941064293:90 }
2023-04-30 12:55:39,274 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=197 (was 342), ProcessCount=170 (was 170), AvailableMemoryMB=10144 (was 10370)