2023-05-04 03:56:37,733 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-04 03:56:38,001 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-04 03:56:38,090 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-04 03:56:38,138 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-04 03:56:38,144 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-04 03:56:38,154 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=113, ProcessCount=168, AvailableMemoryMB=8305
2023-05-04 03:56:38,156 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-04 03:56:38,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-04 03:56:38,260 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100442111:84 , srv10194293:84 , srv1075774841:84 , srv1089162462:84 , srv1091917504:84 , srv1114539562:84 , srv1118778262:84 , srv1150701675:84 , srv1170834224:84 , srv1174028560:84 , srv1180114681:84 , srv1195349783:84 , srv1201428942:84 , srv1202085033:84 , srv1208681438:84 , srv1237745941:84 , srv1256322189:84 , srv1267101282:84 , srv1277684364:84 , srv1299803910:84 , srv1328287388:84 , srv1351362762:84 , srv1359576929:84 , srv1377125774:84 , srv1405014524:84 , srv1446066995:84 , srv1447020904:84 , srv1475595082:84 , srv1479942690:84 , srv1482999076:84 , srv1505494023:84 , srv1526979969:84 , srv154325384:84 , srv1553748991:84 , srv1565776418:84 , srv1572685967:84 , srv1574762311:84 , srv1595644564:84 , srv1614204866:84 , srv1624545528:84 , srv163364623:84 , srv1662717783:84 , srv1663783616:84 , srv1798167188:84 , srv1820270612:84 , srv1824130874:84 , srv1827650853:84 , srv1862414675:84 , srv1875653999:84 , srv1895272138:84 , srv1936349029:84 , srv1948771997:84 , srv1949324989:84 , srv1949735962:84 , srv1951252968:84 , srv1973650175:84 , srv1988555296:84 , srv199302493:84 , srv2031944482:84 , srv2035043935:84 , srv2040986584:84 , srv205327773:84 , srv2053974495:84 , srv2067976970:84 , srv2095316425:84 , srv2103020420:84 , srv2106593970:84 , srv216649262:84 , srv290941511:84 , srv305328373:84 , srv319329053:84 , srv338712853:84 , srv340743450:84 , srv362040843:84 , srv36652311:84 , srv383093980:84 , srv387273694:84 , srv406511611:84 , srv509503104:84 , srv56719893:84 , srv589640342:84 , srv617231332:84 , srv629379794:84 , srv638228764:84 , srv654993223:84 , srv687993412:84 , srv696586127:84 , srv734234330:84 , srv78101769:84 , srv802413970:84 , srv832123423:84 , srv8755055:84 , srv877657207:84 , srv889258302:84 , srv910255450:84 , srv942850396:84 , srv959891645:84 , srv973055276:84 , srv985770283:84 , srv421150285:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-04 03:56:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:38,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338712853=71, srv1202085033=13, srv1973650175=55, srv1614204866=38, srv10194293=1, srv1949735962=53, srv387273694=76, srv629379794=83, srv1505494023=30, srv2106593970=66, srv362040843=73, srv1936349029=50, srv877657207=93, srv1446066995=25, srv2053974495=62, srv985770283=99, srv1447020904=26, srv1624545528=39, srv1150701675=7, srv1526979969=31, srv406511611=77, srv1662717783=41, srv617231332=82, srv889258302=94, srv199302493=57, srv78101769=89, srv163364623=40, srv1180114681=10, srv509503104=79, srv1482999076=29, srv1475595082=27, srv1351362762=21, srv1256322189=16, srv1553748991=33, srv1405014524=24, srv2035043935=59, srv1574762311=36, srv638228764=84, srv973055276=98, srv1572685967=35, srv383093980=75, srv205327773=61, srv1208681438=14, srv2095316425=64, srv1091917504=4, srv2040986584=60, srv687993412=86, srv910255450=95, srv340743450=72, srv1359576929=22, srv1798167188=43, srv216649262=67, srv1114539562=5, srv802413970=90, srv1820270612=44, srv319329053=70, srv1479942690=28, srv832123423=91, srv1895272138=49, srv1824130874=45, srv1277684364=18, srv1862414675=47, srv734234330=88, srv100442111=0, srv589640342=81, srv2067976970=63, srv1174028560=9, srv1875653999=48, srv696586127=87, srv1170834224=8, srv1118778262=6, srv154325384=32, srv1328287388=20, srv654993223=85, srv421150285=78, srv305328373=69, srv290941511=68, srv1951252968=54, srv1595644564=37, srv1195349783=11, srv1377125774=23, srv1988555296=56, srv1089162462=3, srv1949324989=52, srv1663783616=42, srv8755055=92, srv1827650853=46, srv1948771997=51, srv1237745941=15, srv959891645=97, srv1267101282=17, srv2103020420=65, srv2031944482=58, srv1075774841=2, srv1565776418=34, srv36652311=74, srv1299803910=19, srv1201428942=12, srv56719893=80, srv942850396=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:38,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:38,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:38,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-04 03:56:38,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-04 03:56:38,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-04 03:56:38,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-04 03:56:38,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-04 03:56:38,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-04 03:56:38,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-04 03:56:38,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-04 03:56:38,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-04 03:56:38,497 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:38,498 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-04 03:56:38,498 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-04 03:56:38,861 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 473 ms to try 5000 different iterations.  Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18340037943515236. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041034447887787436, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060265033796895566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041034447887787436, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:38,870 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv10194293:81 , srv1948771997:82 , srv654993223:82 , srv696586127:82 , srv734234330:82 , srv942850396:82 , srv1089162462:83 , srv1170834224:83 , srv1405014524:83 , srv1663783616:83 , srv319329053:83 , srv589640342:83 , srv1798167188:84 , srv340743450:84 , srv1075774841:85 , srv1091917504:85 , srv1195349783:85 , srv1201428942:85 , srv1202085033:85 , srv1377125774:85 , srv1447020904:85 , srv1479942690:85 , srv1574762311:85 , srv1875653999:85 , srv1949324989:85 , srv2031944482:85 , srv362040843:85 , srv56719893:85 , srv832123423:85 , srv973055276:85 , srv985770283:85 , srv1150701675:86 , srv1180114681:86 , srv1256322189:86 , srv1359576929:86 , srv1446066995:86 , srv1475595082:86 , srv154325384:86 , srv1595644564:86 , srv1662717783:86 , srv1827650853:86 , srv1862414675:86 , srv1936349029:86 , srv1973650175:86 , srv199302493:86 , srv205327773:86 , srv2053974495:86 , srv2067976970:86 , srv2103020420:86 , srv387273694:86 , srv509503104:86 , srv687993412:86 , srv802413970:86 , srv8755055:86 , srv910255450:86 , srv959891645:86 , srv1174028560:87 , srv1277684364:87 , srv1299803910:87 , srv1328287388:87 , srv1526979969:87 , srv1949735962:87 , srv1988555296:87 , srv2035043935:87 , srv290941511:87 , srv305328373:87 , srv629379794:87 , srv78101769:87 , srv889258302:87 , srv100442111:88 , srv1114539562:88 , srv1118778262:88 , srv1267101282:88 , srv1351362762:88 , srv1482999076:88 , srv1553748991:88 , srv1614204866:88 , srv1820270612:88 , srv1895272138:88 , srv1951252968:88 , srv2095316425:88 , srv216649262:88 , srv338712853:88 , srv383093980:88 , srv406511611:88 , srv617231332:88 , srv638228764:88 , srv877657207:88 , srv1505494023:89 , srv1624545528:89 , srv163364623:89 , srv1824130874:89 , srv2106593970:89 , srv1208681438:90 , srv1572685967:90 , srv36652311:90 , srv1565776418:91 , srv2040986584:91 , srv1237745941:93 , srv421150285:455 }
2023-05-04 03:56:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338712853=71, srv1202085033=13, srv1973650175=55, srv1614204866=38, srv10194293=1, srv1949735962=53, srv387273694=76, srv629379794=83, srv1505494023=30, srv2106593970=66, srv362040843=73, srv1936349029=50, srv877657207=93, srv1446066995=25, srv2053974495=62, srv985770283=99, srv1447020904=26, srv1624545528=39, srv1150701675=7, srv1526979969=31, srv406511611=77, srv1662717783=41, srv617231332=82, srv889258302=94, srv199302493=57, srv78101769=89, srv163364623=40, srv1180114681=10, srv509503104=79, srv1482999076=29, srv1475595082=27, srv1351362762=21, srv1256322189=16, srv1553748991=33, srv1405014524=24, srv2035043935=59, srv1574762311=36, srv638228764=84, srv973055276=98, srv1572685967=35, srv383093980=75, srv205327773=61, srv1208681438=14, srv2095316425=64, srv1091917504=4, srv2040986584=60, srv687993412=86, srv910255450=95, srv340743450=72, srv1359576929=22, srv1798167188=43, srv216649262=67, srv1114539562=5, srv802413970=90, srv1820270612=44, srv319329053=70, srv1479942690=28, srv832123423=91, srv1895272138=49, srv1824130874=45, srv1277684364=18, srv1862414675=47, srv734234330=88, srv100442111=0, srv589640342=81, srv2067976970=63, srv1174028560=9, srv1875653999=48, srv696586127=87, srv1170834224=8, srv1118778262=6, srv154325384=32, srv1328287388=20, srv654993223=85, srv421150285=78, srv305328373=69, srv290941511=68, srv1951252968=54, srv1595644564=37, srv1195349783=11, srv1377125774=23, srv1988555296=56, srv1089162462=3, srv1949324989=52, srv1663783616=42, srv8755055=92, srv1827650853=46, srv1948771997=51, srv1237745941=15, srv959891645=97, srv1267101282=17, srv2103020420=65, srv2031944482=58, srv1075774841=2, srv1565776418=34, srv36652311=74, srv1299803910=19, srv1201428942=12, srv56719893=80, srv942850396=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:38,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:38,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-04 03:56:38,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-04 03:56:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-04 03:56:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-04 03:56:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-04 03:56:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-04 03:56:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-04 03:56:38,984 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041034447887787436, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060265033796895566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041034447887787436, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:38,984 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-04 03:56:38,984 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18336756736184584, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041034447887787436, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060265033796895566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041034447887787436, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-04 03:56:39,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 258 ms to try 5000 different iterations.  Found a solution that moves 2654 regions; Going from a computed imbalance of 0.18336756736184584 to a new imbalance of 0.015237728351438195. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02293796836725213); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03645023358799678, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29488888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02293796836725213); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,177 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1237745941:78 , srv2040986584:80 , srv1572685967:82 , srv163364623:82 , srv2106593970:82 , srv36652311:82 , srv1299803910:83 , srv1482999076:83 , srv1553748991:83 , srv1824130874:83 , srv1949735962:83 , srv1988555296:83 , srv2035043935:83 , srv338712853:83 , srv406511611:83 , srv100442111:84 , srv1114539562:84 , srv1174028560:84 , srv1208681438:84 , srv1267101282:84 , srv1328287388:84 , srv1505494023:84 , srv154325384:84 , srv1565776418:84 , srv1624545528:84 , srv1895272138:84 , srv1951252968:84 , srv2053974495:84 , srv2095316425:84 , srv638228764:84 , srv802413970:84 , srv877657207:84 , srv1075774841:85 , srv1351362762:85 , srv1377125774:85 , srv1526979969:85 , srv1595644564:85 , srv1614204866:85 , srv1662717783:85 , srv1820270612:85 , srv1936349029:85 , srv199302493:85 , srv216649262:85 , srv290941511:85 , srv305328373:85 , srv362040843:85 , srv509503104:85 , srv56719893:85 , srv617231332:85 , srv8755055:85 , srv973055276:85 , srv1150701675:86 , srv1180114681:86 , srv1256322189:86 , srv1277684364:86 , srv1446066995:86 , srv1862414675:86 , srv205327773:86 , srv2067976970:86 , srv383093980:86 , srv687993412:86 , srv78101769:86 , srv832123423:86 , srv889258302:86 , srv1089162462:87 , srv1118778262:87 , srv1195349783:87 , srv1201428942:87 , srv1202085033:87 , srv1447020904:87 , srv1475595082:87 , srv1479942690:87 , srv1827650853:87 , srv1949324989:87 , srv1973650175:87 , srv2031944482:87 , srv2103020420:87 , srv387273694:87 , srv589640342:87 , srv629379794:87 , srv959891645:87 , srv985770283:87 , srv1091917504:88 , srv1359576929:88 , srv1405014524:88 , srv1663783616:88 , srv1798167188:88 , srv319329053:88 , srv910255450:88 , srv1170834224:89 , srv1574762311:89 , srv1875653999:89 , srv340743450:89 , srv734234330:89 , srv942850396:89 , srv10194293:91 , srv696586127:91 , srv1948771997:92 , srv654993223:93 , srv421150285:523 }
2023-05-04 03:56:39,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:39,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338712853=71, srv1202085033=13, srv1973650175=55, srv1614204866=38, srv10194293=1, srv1949735962=53, srv387273694=76, srv629379794=83, srv1505494023=30, srv2106593970=66, srv362040843=73, srv1936349029=50, srv877657207=93, srv1446066995=25, srv2053974495=62, srv985770283=99, srv1447020904=26, srv1624545528=39, srv1150701675=7, srv1526979969=31, srv406511611=77, srv1662717783=41, srv617231332=82, srv889258302=94, srv199302493=57, srv78101769=89, srv163364623=40, srv1180114681=10, srv509503104=79, srv1482999076=29, srv1475595082=27, srv1351362762=21, srv1256322189=16, srv1553748991=33, srv1405014524=24, srv2035043935=59, srv1574762311=36, srv638228764=84, srv973055276=98, srv1572685967=35, srv383093980=75, srv205327773=61, srv1208681438=14, srv2095316425=64, srv1091917504=4, srv2040986584=60, srv687993412=86, srv910255450=95, srv340743450=72, srv1359576929=22, srv1798167188=43, srv216649262=67, srv1114539562=5, srv802413970=90, srv1820270612=44, srv319329053=70, srv1479942690=28, srv832123423=91, srv1895272138=49, srv1824130874=45, srv1277684364=18, srv1862414675=47, srv734234330=88, srv100442111=0, srv589640342=81, srv2067976970=63, srv1174028560=9, srv1875653999=48, srv696586127=87, srv1170834224=8, srv1118778262=6, srv154325384=32, srv1328287388=20, srv654993223=85, srv421150285=78, srv305328373=69, srv290941511=68, srv1951252968=54, srv1595644564=37, srv1195349783=11, srv1377125774=23, srv1988555296=56, srv1089162462=3, srv1949324989=52, srv1663783616=42, srv8755055=92, srv1827650853=46, srv1948771997=51, srv1237745941=15, srv959891645=97, srv1267101282=17, srv2103020420=65, srv2031944482=58, srv1075774841=2, srv1565776418=34, srv36652311=74, srv1299803910=19, srv1201428942=12, srv56719893=80, srv942850396=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:39,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:39,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-04 03:56:39,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-04 03:56:39,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-04 03:56:39,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-04 03:56:39,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-04 03:56:39,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-04 03:56:39,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-04 03:56:39,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-04 03:56:39,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-04 03:56:39,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-04 03:56:39,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-04 03:56:39,293 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02293796836725213); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03645023358799678, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02293796836725213); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,293 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-04 03:56:39,294 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0152191446462407, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02293796836725213); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03645023358799678, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02293796836725213); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-04 03:56:39,449 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 229 ms to try 5000 different iterations.  Found a solution that moves 1038 regions; Going from a computed imbalance of 0.0152191446462407 to a new imbalance of 5.320938867747961E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008361158185638068); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.11533333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008361158185638068); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,451 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1565776418:81 , srv654993223:81 , srv1118778262:83 , srv1359576929:83 , srv1475595082:83 , srv1505494023:83 , srv1614204866:83 , srv1875653999:83 , srv1948771997:83 , srv1951252968:83 , srv910255450:83 , srv959891645:83 , srv100442111:84 , srv1114539562:84 , srv1150701675:84 , srv1208681438:84 , srv1277684364:84 , srv1479942690:84 , srv1574762311:84 , srv1624545528:84 , srv1820270612:84 , srv1895272138:84 , srv1973650175:84 , srv2103020420:84 , srv383093980:84 , srv387273694:84 , srv629379794:84 , srv78101769:84 , srv889258302:84 , srv1091917504:85 , srv1170834224:85 , srv1174028560:85 , srv1180114681:85 , srv1201428942:85 , srv1237745941:85 , srv1267101282:85 , srv1328287388:85 , srv1351362762:85 , srv1447020904:85 , srv1572685967:85 , srv1798167188:85 , srv1949324989:85 , srv2031944482:85 , srv205327773:85 , srv2067976970:85 , srv2095316425:85 , srv305328373:85 , srv340743450:85 , srv617231332:85 , srv696586127:85 , srv10194293:86 , srv1195349783:86 , srv1202085033:86 , srv1256322189:86 , srv1377125774:86 , srv1405014524:86 , srv1446066995:86 , srv1553748991:86 , srv1595644564:86 , srv1662717783:86 , srv1663783616:86 , srv1824130874:86 , srv1827650853:86 , srv1862414675:86 , srv199302493:86 , srv2035043935:86 , srv2053974495:86 , srv2106593970:86 , srv290941511:86 , srv338712853:86 , srv36652311:86 , srv406511611:86 , srv509503104:86 , srv638228764:86 , srv687993412:86 , srv8755055:86 , srv877657207:86 , srv942850396:86 , srv985770283:86 , srv1299803910:87 , srv1482999076:87 , srv1526979969:87 , srv154325384:87 , srv163364623:87 , srv1949735962:87 , srv2040986584:87 , srv216649262:87 , srv319329053:87 , srv362040843:87 , srv589640342:87 , srv802413970:87 , srv832123423:87 , srv973055276:87 , srv1089162462:88 , srv1936349029:88 , srv1988555296:88 , srv56719893:88 , srv734234330:88 , srv1075774841:89 , srv421150285:554 }
2023-05-04 03:56:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338712853=71, srv1202085033=13, srv1973650175=55, srv1614204866=38, srv10194293=1, srv1949735962=53, srv387273694=76, srv629379794=83, srv1505494023=30, srv2106593970=66, srv362040843=73, srv1936349029=50, srv877657207=93, srv1446066995=25, srv2053974495=62, srv985770283=99, srv1447020904=26, srv1624545528=39, srv1150701675=7, srv1526979969=31, srv406511611=77, srv1662717783=41, srv617231332=82, srv889258302=94, srv199302493=57, srv78101769=89, srv163364623=40, srv1180114681=10, srv509503104=79, srv1482999076=29, srv1475595082=27, srv1351362762=21, srv1256322189=16, srv1553748991=33, srv1405014524=24, srv2035043935=59, srv1574762311=36, srv638228764=84, srv973055276=98, srv1572685967=35, srv383093980=75, srv205327773=61, srv1208681438=14, srv2095316425=64, srv1091917504=4, srv2040986584=60, srv687993412=86, srv910255450=95, srv340743450=72, srv1359576929=22, srv1798167188=43, srv216649262=67, srv1114539562=5, srv802413970=90, srv1820270612=44, srv319329053=70, srv1479942690=28, srv832123423=91, srv1895272138=49, srv1824130874=45, srv1277684364=18, srv1862414675=47, srv734234330=88, srv100442111=0, srv589640342=81, srv2067976970=63, srv1174028560=9, srv1875653999=48, srv696586127=87, srv1170834224=8, srv1118778262=6, srv154325384=32, srv1328287388=20, srv654993223=85, srv421150285=78, srv305328373=69, srv290941511=68, srv1951252968=54, srv1595644564=37, srv1195349783=11, srv1377125774=23, srv1988555296=56, srv1089162462=3, srv1949324989=52, srv1663783616=42, srv8755055=92, srv1827650853=46, srv1948771997=51, srv1237745941=15, srv959891645=97, srv1267101282=17, srv2103020420=65, srv2031944482=58, srv1075774841=2, srv1565776418=34, srv36652311=74, srv1299803910=19, srv1201428942=12, srv56719893=80, srv942850396=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:39,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:39,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-04 03:56:39,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-04 03:56:39,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-04 03:56:39,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-04 03:56:39,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-04 03:56:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-04 03:56:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-04 03:56:39,552 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008361158185638068); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008361158185638068); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,552 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-04 03:56:39,553 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.248256554277725E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008361158185638068); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008361158185638068); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-04 03:56:39,765 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 283 ms to try 5000 different iterations.  Found a solution that moves 283 regions; Going from a computed imbalance of 5.248256554277725E-4 to a new imbalance of 2.0419943753075508E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,766 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1827650853:82 , srv216649262:82 , srv1075774841:83 , srv1118778262:83 , srv1256322189:83 , srv1936349029:83 , srv734234330:83 , srv10194293:84 , srv1091917504:84 , srv1180114681:84 , srv1195349783:84 , srv1201428942:84 , srv1202085033:84 , srv1208681438:84 , srv1351362762:84 , srv1446066995:84 , srv1482999076:84 , srv1526979969:84 , srv1574762311:84 , srv163364623:84 , srv1663783616:84 , srv1824130874:84 , srv1988555296:84 , srv2040986584:84 , srv2106593970:84 , srv290941511:84 , srv319329053:84 , srv340743450:84 , srv36652311:84 , srv383093980:84 , srv387273694:84 , srv56719893:84 , srv617231332:84 , srv629379794:84 , srv638228764:84 , srv687993412:84 , srv696586127:84 , srv832123423:84 , srv877657207:84 , srv942850396:84 , srv985770283:84 , srv1089162462:85 , srv1170834224:85 , srv1277684364:85 , srv1299803910:85 , srv1359576929:85 , srv1405014524:85 , srv1447020904:85 , srv1479942690:85 , srv154325384:85 , srv1572685967:85 , srv1595644564:85 , srv1624545528:85 , srv1662717783:85 , srv1798167188:85 , srv1820270612:85 , srv1862414675:85 , srv1875653999:85 , srv1948771997:85 , srv1949324989:85 , srv1949735962:85 , srv1973650175:85 , srv199302493:85 , srv2031944482:85 , srv2067976970:85 , srv2095316425:85 , srv2103020420:85 , srv362040843:85 , srv406511611:85 , srv509503104:85 , srv589640342:85 , srv802413970:85 , srv8755055:85 , srv889258302:85 , srv910255450:85 , srv973055276:85 , srv100442111:86 , srv1114539562:86 , srv1150701675:86 , srv1174028560:86 , srv1267101282:86 , srv1328287388:86 , srv1377125774:86 , srv1475595082:86 , srv1505494023:86 , srv1553748991:86 , srv1565776418:86 , srv1614204866:86 , srv1895272138:86 , srv2035043935:86 , srv205327773:86 , srv2053974495:86 , srv305328373:86 , srv338712853:86 , srv654993223:86 , srv78101769:86 , srv959891645:86 , srv1237745941:87 , srv1951252968:87 , srv421150285:610 }
2023-05-04 03:56:39,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:39,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338712853=71, srv1202085033=13, srv1973650175=55, srv1614204866=38, srv10194293=1, srv1949735962=53, srv387273694=76, srv629379794=83, srv1505494023=30, srv2106593970=66, srv362040843=73, srv1936349029=50, srv877657207=93, srv1446066995=25, srv2053974495=62, srv985770283=99, srv1447020904=26, srv1624545528=39, srv1150701675=7, srv1526979969=31, srv406511611=77, srv1662717783=41, srv617231332=82, srv889258302=94, srv199302493=57, srv78101769=89, srv163364623=40, srv1180114681=10, srv509503104=79, srv1482999076=29, srv1475595082=27, srv1351362762=21, srv1256322189=16, srv1553748991=33, srv1405014524=24, srv2035043935=59, srv1574762311=36, srv638228764=84, srv973055276=98, srv1572685967=35, srv383093980=75, srv205327773=61, srv1208681438=14, srv2095316425=64, srv1091917504=4, srv2040986584=60, srv687993412=86, srv910255450=95, srv340743450=72, srv1359576929=22, srv1798167188=43, srv216649262=67, srv1114539562=5, srv802413970=90, srv1820270612=44, srv319329053=70, srv1479942690=28, srv832123423=91, srv1895272138=49, srv1824130874=45, srv1277684364=18, srv1862414675=47, srv734234330=88, srv100442111=0, srv589640342=81, srv2067976970=63, srv1174028560=9, srv1875653999=48, srv696586127=87, srv1170834224=8, srv1118778262=6, srv154325384=32, srv1328287388=20, srv654993223=85, srv421150285=78, srv305328373=69, srv290941511=68, srv1951252968=54, srv1595644564=37, srv1195349783=11, srv1377125774=23, srv1988555296=56, srv1089162462=3, srv1949324989=52, srv1663783616=42, srv8755055=92, srv1827650853=46, srv1948771997=51, srv1237745941=15, srv959891645=97, srv1267101282=17, srv2103020420=65, srv2031944482=58, srv1075774841=2, srv1565776418=34, srv36652311=74, srv1299803910=19, srv1201428942=12, srv56719893=80, srv942850396=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:39,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:39,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:39,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:39,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:39,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-04 03:56:39,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-04 03:56:39,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-04 03:56:39,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-04 03:56:39,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-04 03:56:39,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-04 03:56:39,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-04 03:56:39,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-04 03:56:39,872 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8438335399310903E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,872 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1827650853:82 , srv216649262:82 , srv1075774841:83 , srv1118778262:83 , srv1256322189:83 , srv1936349029:83 , srv734234330:83 , srv10194293:84 , srv1091917504:84 , srv1180114681:84 , srv1195349783:84 , srv1201428942:84 , srv1202085033:84 , srv1208681438:84 , srv1351362762:84 , srv1446066995:84 , srv1482999076:84 , srv1526979969:84 , srv1574762311:84 , srv163364623:84 , srv1663783616:84 , srv1824130874:84 , srv1988555296:84 , srv2040986584:84 , srv2106593970:84 , srv290941511:84 , srv319329053:84 , srv340743450:84 , srv36652311:84 , srv383093980:84 , srv387273694:84 , srv56719893:84 , srv617231332:84 , srv629379794:84 , srv638228764:84 , srv687993412:84 , srv696586127:84 , srv832123423:84 , srv877657207:84 , srv942850396:84 , srv985770283:84 , srv1089162462:85 , srv1170834224:85 , srv1277684364:85 , srv1299803910:85 , srv1359576929:85 , srv1405014524:85 , srv1447020904:85 , srv1479942690:85 , srv154325384:85 , srv1572685967:85 , srv1595644564:85 , srv1624545528:85 , srv1662717783:85 , srv1798167188:85 , srv1820270612:85 , srv1862414675:85 , srv1875653999:85 , srv1948771997:85 , srv1949324989:85 , srv1949735962:85 , srv1973650175:85 , srv199302493:85 , srv2031944482:85 , srv2067976970:85 , srv2095316425:85 , srv2103020420:85 , srv362040843:85 , srv406511611:85 , srv509503104:85 , srv589640342:85 , srv802413970:85 , srv8755055:85 , srv889258302:85 , srv910255450:85 , srv973055276:85 , srv100442111:86 , srv1114539562:86 , srv1150701675:86 , srv1174028560:86 , srv1267101282:86 , srv1328287388:86 , srv1377125774:86 , srv1475595082:86 , srv1505494023:86 , srv1553748991:86 , srv1565776418:86 , srv1614204866:86 , srv1895272138:86 , srv2035043935:86 , srv205327773:86 , srv2053974495:86 , srv305328373:86 , srv338712853:86 , srv654993223:86 , srv78101769:86 , srv959891645:86 , srv1237745941:87 , srv1951252968:87 , srv421150285:610 }
2023-05-04 03:56:39,899 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=121 (was 113) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=7838 (was 8305)
2023-05-04 03:56:39,906 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=168, AvailableMemoryMB=7837
2023-05-04 03:56:39,907 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-04 03:56:39,907 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-04 03:56:39,916 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1061044780:84 , srv1144270687:84 , srv1147947888:84 , srv1162165783:84 , srv1217135753:84 , srv1275113921:84 , srv1477620358:84 , srv1495400438:84 , srv1651629463:84 , srv1689696723:84 , srv175268781:84 , srv1806607722:84 , srv1815184196:84 , srv1820500778:84 , srv1854173513:84 , srv1899743043:84 , srv1903609465:84 , srv191637398:84 , srv1949467706:84 , srv1970922935:84 , srv215728902:84 , srv307383852:84 , srv557495675:84 , srv605113114:84 , srv609031800:84 , srv704044216:84 , srv707963538:84 , srv711938481:84 , srv943211938:84 , srv1179223948:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-04 03:56:39,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:56:39,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1495400438=8, srv1217135753=5, srv711938481=28, srv704044216=26, srv215728902=21, srv307383852=22, srv1144270687=1, srv1651629463=9, srv1903609465=17, srv943211938=29, srv1854173513=15, srv1061044780=0, srv1162165783=3, srv1689696723=10, srv1820500778=14, srv1899743043=16, srv609031800=25, srv1806607722=12, srv1147947888=2, srv1970922935=20, srv191637398=18, srv1949467706=19, srv557495675=23, srv1815184196=13, srv1477620358=7, srv1275113921=6, srv1179223948=4, srv175268781=11, srv605113114=24, srv707963538=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-04 03:56:39,938 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1107007932191775, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:56:39,939 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912541857849001, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1107007932191775, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-04 03:58:39,929 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9938526 different iterations.  Found a solution that moves 1831 regions; Going from a computed imbalance of 0.9912541857849001 to a new imbalance of 4.2736453424534666E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6781481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:58:39,934 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1061044780:90 , srv1144270687:90 , srv1147947888:90 , srv1162165783:90 , srv1179223948:90 , srv1217135753:90 , srv1275113921:90 , srv1477620358:90 , srv1495400438:90 , srv1651629463:90 , srv1689696723:90 , srv175268781:90 , srv1806607722:90 , srv1815184196:90 , srv1820500778:90 , srv1854173513:90 , srv1899743043:90 , srv1903609465:90 , srv191637398:90 , srv1949467706:90 , srv1970922935:90 , srv215728902:90 , srv307383852:90 , srv557495675:90 , srv605113114:90 , srv609031800:90 , srv704044216:90 , srv707963538:90 , srv711938481:90 , srv943211938:90 }
2023-05-04 03:58:39,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-04 03:58:39,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1495400438=8, srv1217135753=5, srv711938481=28, srv704044216=26, srv215728902=21, srv307383852=22, srv1144270687=1, srv1651629463=9, srv1903609465=17, srv943211938=29, srv1854173513=15, srv1061044780=0, srv1162165783=3, srv1689696723=10, srv1820500778=14, srv1899743043=16, srv609031800=25, srv1806607722=12, srv1147947888=2, srv1970922935=20, srv191637398=18, srv1949467706=19, srv557495675=23, srv1815184196=13, srv1477620358=7, srv1275113921=6, srv1179223948=4, srv175268781=11, srv605113114=24, srv707963538=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-04 03:58:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-04 03:58:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-04 03:58:39,950 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-04 03:58:39,950 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1061044780:90 , srv1144270687:90 , srv1147947888:90 , srv1162165783:90 , srv1179223948:90 , srv1217135753:90 , srv1275113921:90 , srv1477620358:90 , srv1495400438:90 , srv1651629463:90 , srv1689696723:90 , srv175268781:90 , srv1806607722:90 , srv1815184196:90 , srv1820500778:90 , srv1854173513:90 , srv1899743043:90 , srv1903609465:90 , srv191637398:90 , srv1949467706:90 , srv1970922935:90 , srv215728902:90 , srv307383852:90 , srv557495675:90 , srv605113114:90 , srv609031800:90 , srv704044216:90 , srv707963538:90 , srv711938481:90 , srv943211938:90 }
2023-05-04 03:58:39,961 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=156 (was 121) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 168), AvailableMemoryMB=7836 (was 7837)