2023-05-20 03:53:25,896 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-20 03:53:26,215 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-20 03:53:26,325 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-20 03:53:26,383 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-20 03:53:26,391 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-20 03:53:26,404 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=389, ProcessCount=176, AvailableMemoryMB=8354
2023-05-20 03:53:26,406 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-20 03:53:26,406 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-20 03:53:26,556 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005514051:84 , srv1084362363:84 , srv1086547822:84 , srv1092745480:84 , srv1132303614:84 , srv1174091028:84 , srv1187552366:84 , srv1209025083:84 , srv1241572002:84 , srv1242014112:84 , srv1247015677:84 , srv125634843:84 , srv1320904893:84 , srv1326987750:84 , srv1395040171:84 , srv1457341107:84 , srv1485405387:84 , srv1530652214:84 , srv1531237906:84 , srv1532465721:84 , srv1540375263:84 , srv1544940064:84 , srv1547865263:84 , srv1571296806:84 , srv1597247871:84 , srv1604353144:84 , srv1611649607:84 , srv1632933080:84 , srv1650790998:84 , srv1675030529:84 , srv1688730175:84 , srv1704833961:84 , srv1706463030:84 , srv1714639493:84 , srv1780669202:84 , srv1799781224:84 , srv1833125715:84 , srv1834781165:84 , srv1846109764:84 , srv1906651700:84 , srv1914371605:84 , srv1925442669:84 , srv1976545914:84 , srv2011221269:84 , srv2036954215:84 , srv2037591242:84 , srv2071828250:84 , srv2086355439:84 , srv2112218724:84 , srv2129601970:84 , srv2134264019:84 , srv2140068891:84 , srv2143561448:84 , srv244699492:84 , srv27649337:84 , srv278240069:84 , srv30249479:84 , srv331232902:84 , srv337567916:84 , srv340041161:84 , srv385018931:84 , srv4244085:84 , srv432961836:84 , srv435281394:84 , srv441611867:84 , srv449496579:84 , srv472314018:84 , srv514350414:84 , srv515009166:84 , srv51524327:84 , srv592444309:84 , srv618175898:84 , srv639090973:84 , srv646642320:84 , srv661318165:84 , srv698336798:84 , srv703466415:84 , srv760357794:84 , srv772006989:84 , srv794143127:84 , srv812019433:84 , srv83647784:84 , srv839295578:84 , srv848547324:84 , srv859643511:84 , srv875789698:84 , srv887492430:84 , srv888514130:84 , srv888752425:84 , srv893493858:84 , srv894052468:84 , srv89479825:84 , srv899704089:84 , srv912371304:84 , srv930061792:84 , srv958614823:84 , srv977225643:84 , srv980100006:84 , srv997045154:84 , srv1539721645:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-20 03:53:26,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:26,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:26,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:26,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:26,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:26,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:26,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:26,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:26,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:26,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:26,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:26,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:26,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:26,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:26,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:26,854 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:26,854 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-20 03:53:26,854 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-20 03:53:27,218 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 529 ms to try 5000 different iterations.  Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18728564962679192. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04214624861838321, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05386727672421822, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04214624861838321, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18266666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24833333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:27,252 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2071828250:80 , srv839295578:80 , srv244699492:81 , srv331232902:81 , srv337567916:81 , srv958614823:81 , srv1976545914:82 , srv4244085:82 , srv618175898:82 , srv812019433:82 , srv83647784:82 , srv893493858:82 , srv1084362363:83 , srv1611649607:83 , srv1675030529:83 , srv441611867:83 , srv592444309:83 , srv888752425:83 , srv980100006:83 , srv1092745480:84 , srv125634843:84 , srv1531237906:84 , srv1544940064:84 , srv1604353144:84 , srv1688730175:84 , srv2036954215:84 , srv2037591242:84 , srv2086355439:84 , srv385018931:84 , srv661318165:84 , srv859643511:84 , srv912371304:84 , srv977225643:84 , srv997045154:84 , srv1187552366:85 , srv1650790998:85 , srv1714639493:85 , srv1906651700:85 , srv2129601970:85 , srv2140068891:85 , srv2143561448:85 , srv27649337:85 , srv30249479:85 , srv515009166:85 , srv646642320:85 , srv703466415:85 , srv772006989:85 , srv887492430:85 , srv1247015677:86 , srv1326987750:86 , srv1532465721:86 , srv1540375263:86 , srv1597247871:86 , srv1704833961:86 , srv1846109764:86 , srv278240069:86 , srv875789698:86 , srv894052468:86 , srv89479825:86 , srv899704089:86 , srv1086547822:87 , srv1174091028:87 , srv1242014112:87 , srv1457341107:87 , srv1530652214:87 , srv1547865263:87 , srv1833125715:87 , srv435281394:87 , srv514350414:87 , srv794143127:87 , srv888514130:87 , srv1632933080:88 , srv1706463030:88 , srv1799781224:88 , srv2011221269:88 , srv51524327:88 , srv1395040171:89 , srv1834781165:89 , srv1925442669:89 , srv2112218724:89 , srv2134264019:89 , srv449496579:89 , srv1485405387:90 , srv639090973:90 , srv1571296806:91 , srv1780669202:91 , srv340041161:91 , srv432961836:91 , srv848547324:91 , srv1241572002:92 , srv698336798:92 , srv930061792:92 , srv1132303614:93 , srv1320904893:93 , srv472314018:93 , srv1005514051:94 , srv1209025083:94 , srv1914371605:94 , srv760357794:94 , srv1539721645:464 }
2023-05-20 03:53:27,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:27,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:27,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:27,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:27,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:27,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:27,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:27,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:27,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:27,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:27,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:27,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:27,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:27,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:27,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:27,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:27,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:27,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:27,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04214624861838321, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05386727672421822, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04214624861838321, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18266666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24833333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:27,487 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-20 03:53:27,488 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1872528235491861, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04214624861838321, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05386727672421822, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04214624861838321, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18266666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24833333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-20 03:53:27,881 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 545 ms to try 5000 different iterations.  Found a solution that moves 2661 regions; Going from a computed imbalance of 0.1872528235491861 to a new imbalance of 0.015003028136598762. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023409900561936193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03379478572335593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2956666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023409900561936193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:27,893 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005514051:76 , srv472314018:76 , srv1914371605:77 , srv1241572002:78 , srv1132303614:79 , srv1209025083:79 , srv1320904893:79 , srv1485405387:80 , srv340041161:80 , srv760357794:80 , srv930061792:80 , srv698336798:81 , srv1395040171:82 , srv1547865263:82 , srv2112218724:82 , srv432961836:82 , srv639090973:82 , srv848547324:82 , srv1632933080:83 , srv1833125715:83 , srv51524327:83 , srv1086547822:84 , srv1457341107:84 , srv1571296806:84 , srv1704833961:84 , srv1799781224:84 , srv2011221269:84 , srv2134264019:84 , srv875789698:84 , srv894052468:84 , srv1187552366:85 , srv1532465721:85 , srv1540375263:85 , srv1706463030:85 , srv1780669202:85 , srv2129601970:85 , srv27649337:85 , srv435281394:85 , srv449496579:85 , srv794143127:85 , srv888514130:85 , srv89479825:85 , srv1174091028:86 , srv1242014112:86 , srv1247015677:86 , srv1326987750:86 , srv1604353144:86 , srv1611649607:86 , srv1650790998:86 , srv1714639493:86 , srv1834781165:86 , srv1846109764:86 , srv1925442669:86 , srv2086355439:86 , srv514350414:86 , srv703466415:86 , srv772006989:86 , srv125634843:87 , srv1530652214:87 , srv1675030529:87 , srv1688730175:87 , srv2140068891:87 , srv2143561448:87 , srv646642320:87 , srv887492430:87 , srv980100006:87 , srv997045154:87 , srv1531237906:88 , srv1906651700:88 , srv2036954215:88 , srv385018931:88 , srv618175898:88 , srv661318165:88 , srv812019433:88 , srv899704089:88 , srv977225643:88 , srv1092745480:89 , srv1976545914:89 , srv2037591242:89 , srv278240069:89 , srv30249479:89 , srv331232902:89 , srv337567916:89 , srv515009166:89 , srv592444309:89 , srv859643511:89 , srv912371304:89 , srv1544940064:90 , srv1597247871:90 , srv441611867:90 , srv83647784:90 , srv893493858:90 , srv958614823:90 , srv1084362363:91 , srv2071828250:91 , srv244699492:91 , srv4244085:91 , srv888752425:92 , srv839295578:93 , srv1539721645:518 }
2023-05-20 03:53:27,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:27,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:27,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:27,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:27,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:27,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:27,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:27,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:27,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:27,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:27,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:27,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:27,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:27,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:27,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:27,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:27,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:28,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023409900561936193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03379478572335593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023409900561936193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:28,079 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-20 03:53:28,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014984395416353645, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023409900561936193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03379478572335593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023409900561936193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-20 03:53:28,421 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 452 ms to try 5000 different iterations.  Found a solution that moves 1006 regions; Going from a computed imbalance of 0.014984395416353645 to a new imbalance of 7.59513088885364E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.11177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:28,423 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1571296806:82 , srv1597247871:82 , srv1780669202:82 , srv1925442669:82 , srv278240069:82 , srv1084362363:83 , srv1834781165:83 , srv760357794:83 , srv888752425:83 , srv1174091028:84 , srv1242014112:84 , srv1530652214:84 , srv1544940064:84 , srv2140068891:84 , srv2143561448:84 , srv244699492:84 , srv30249479:84 , srv449496579:84 , srv514350414:84 , srv515009166:84 , srv698336798:84 , srv899704089:84 , srv912371304:84 , srv1092745480:85 , srv1209025083:85 , srv1320904893:85 , srv1531237906:85 , srv1706463030:85 , srv1799781224:85 , srv1906651700:85 , srv1914371605:85 , srv2011221269:85 , srv2036954215:85 , srv2037591242:85 , srv2134264019:85 , srv385018931:85 , srv4244085:85 , srv435281394:85 , srv441611867:85 , srv592444309:85 , srv639090973:85 , srv83647784:85 , srv839295578:85 , srv848547324:85 , srv859643511:85 , srv888514130:85 , srv893493858:85 , srv977225643:85 , srv1086547822:86 , srv1132303614:86 , srv1247015677:86 , srv125634843:86 , srv1326987750:86 , srv1457341107:86 , srv1532465721:86 , srv1540375263:86 , srv1632933080:86 , srv1846109764:86 , srv2071828250:86 , srv2112218724:86 , srv340041161:86 , srv432961836:86 , srv51524327:86 , srv646642320:86 , srv661318165:86 , srv794143127:86 , srv887492430:86 , srv89479825:86 , srv930061792:86 , srv958614823:86 , srv1005514051:87 , srv1395040171:87 , srv1604353144:87 , srv1650790998:87 , srv1675030529:87 , srv1688730175:87 , srv1704833961:87 , srv1714639493:87 , srv1833125715:87 , srv1976545914:87 , srv331232902:87 , srv618175898:87 , srv703466415:87 , srv772006989:87 , srv812019433:87 , srv997045154:87 , srv1187552366:88 , srv1241572002:88 , srv1485405387:88 , srv2086355439:88 , srv2129601970:88 , srv27649337:88 , srv337567916:88 , srv875789698:88 , srv894052468:88 , srv980100006:88 , srv1547865263:89 , srv1611649607:89 , srv472314018:89 , srv1539721645:526 }
2023-05-20 03:53:28,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:28,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:28,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:28,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:28,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:28,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:28,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:28,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:28,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:28,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:28,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:28,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:28,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:28,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:28,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:28,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:28,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:28,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:28,624 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-20 03:53:28,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.524689263274587E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-20 03:53:28,975 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 489 ms to try 5000 different iterations.  Found a solution that moves 211 regions; Going from a computed imbalance of 7.524689263274587E-4 to a new imbalance of 3.39449909681629E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.023444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:28,975 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1906651700:83 , srv432961836:83 , srv472314018:83 , srv1005514051:84 , srv1092745480:84 , srv1132303614:84 , srv1187552366:84 , srv1209025083:84 , srv1241572002:84 , srv1247015677:84 , srv1326987750:84 , srv1395040171:84 , srv1485405387:84 , srv1530652214:84 , srv1544940064:84 , srv1547865263:84 , srv1597247871:84 , srv1611649607:84 , srv1650790998:84 , srv1688730175:84 , srv1706463030:84 , srv1714639493:84 , srv1780669202:84 , srv1834781165:84 , srv1846109764:84 , srv1976545914:84 , srv2037591242:84 , srv2086355439:84 , srv2129601970:84 , srv2134264019:84 , srv27649337:84 , srv30249479:84 , srv4244085:84 , srv435281394:84 , srv441611867:84 , srv449496579:84 , srv515009166:84 , srv646642320:84 , srv703466415:84 , srv772006989:84 , srv794143127:84 , srv839295578:84 , srv848547324:84 , srv859643511:84 , srv887492430:84 , srv888514130:84 , srv888752425:84 , srv899704089:84 , srv930061792:84 , srv980100006:84 , srv997045154:84 , srv1084362363:85 , srv1086547822:85 , srv1174091028:85 , srv1242014112:85 , srv125634843:85 , srv1320904893:85 , srv1457341107:85 , srv1531237906:85 , srv1532465721:85 , srv1540375263:85 , srv1571296806:85 , srv1604353144:85 , srv1632933080:85 , srv1675030529:85 , srv1704833961:85 , srv1799781224:85 , srv1833125715:85 , srv1925442669:85 , srv2011221269:85 , srv2036954215:85 , srv2071828250:85 , srv2112218724:85 , srv278240069:85 , srv331232902:85 , srv337567916:85 , srv340041161:85 , srv385018931:85 , srv514350414:85 , srv51524327:85 , srv592444309:85 , srv618175898:85 , srv639090973:85 , srv661318165:85 , srv698336798:85 , srv760357794:85 , srv812019433:85 , srv875789698:85 , srv893493858:85 , srv894052468:85 , srv89479825:85 , srv912371304:85 , srv958614823:85 , srv977225643:85 , srv1914371605:86 , srv2140068891:86 , srv2143561448:86 , srv244699492:86 , srv83647784:86 , srv1539721645:634 }
2023-05-20 03:53:28,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:28,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:28,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:28,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:29,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:29,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:29,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:29,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:29,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:29,115 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:29,116 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-20 03:53:29,116 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.246753738991438E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-20 03:53:29,384 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations.  Found a solution that moves 17 regions; Going from a computed imbalance of 3.246753738991438E-5 to a new imbalance of 7.506794044531634E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.001888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:29,385 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv337567916:83 , srv661318165:83 , srv83647784:83 , srv875789698:83 , srv894052468:83 , srv1084362363:84 , srv1086547822:84 , srv1092745480:84 , srv1132303614:84 , srv1174091028:84 , srv1187552366:84 , srv1209025083:84 , srv1241572002:84 , srv1242014112:84 , srv1247015677:84 , srv125634843:84 , srv1320904893:84 , srv1326987750:84 , srv1395040171:84 , srv1457341107:84 , srv1485405387:84 , srv1530652214:84 , srv1531237906:84 , srv1532465721:84 , srv1540375263:84 , srv1544940064:84 , srv1547865263:84 , srv1571296806:84 , srv1597247871:84 , srv1604353144:84 , srv1611649607:84 , srv1632933080:84 , srv1650790998:84 , srv1675030529:84 , srv1688730175:84 , srv1704833961:84 , srv1706463030:84 , srv1714639493:84 , srv1780669202:84 , srv1799781224:84 , srv1833125715:84 , srv1834781165:84 , srv1846109764:84 , srv1914371605:84 , srv1925442669:84 , srv1976545914:84 , srv2011221269:84 , srv2036954215:84 , srv2037591242:84 , srv2071828250:84 , srv2086355439:84 , srv2112218724:84 , srv2129601970:84 , srv2134264019:84 , srv2140068891:84 , srv2143561448:84 , srv244699492:84 , srv27649337:84 , srv278240069:84 , srv30249479:84 , srv331232902:84 , srv340041161:84 , srv385018931:84 , srv4244085:84 , srv432961836:84 , srv441611867:84 , srv449496579:84 , srv514350414:84 , srv515009166:84 , srv51524327:84 , srv592444309:84 , srv618175898:84 , srv639090973:84 , srv646642320:84 , srv698336798:84 , srv703466415:84 , srv760357794:84 , srv772006989:84 , srv794143127:84 , srv812019433:84 , srv839295578:84 , srv848547324:84 , srv859643511:84 , srv887492430:84 , srv888752425:84 , srv893493858:84 , srv89479825:84 , srv899704089:84 , srv912371304:84 , srv930061792:84 , srv958614823:84 , srv977225643:84 , srv980100006:84 , srv997045154:84 , srv1005514051:85 , srv1906651700:85 , srv435281394:85 , srv472314018:85 , srv888514130:85 , srv1539721645:684 }
2023-05-20 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1242014112=9, srv331232902=58, srv639090973=73, srv888514130=88, srv618175898=72, srv515009166=69, srv2112218724=49, srv125634843=11, srv514350414=68, srv848547324=84, srv441611867=65, srv1914371605=41, srv887492430=87, srv958614823=96, srv661318165=75, srv2134264019=51, srv1005514051=0, srv432961836=63, srv1834781165=38, srv1611649607=27, srv472314018=67, srv1132303614=4, srv1780669202=35, srv385018931=61, srv1650790998=29, srv1833125715=37, srv980100006=98, srv930061792=95, srv51524327=70, srv1846109764=39, srv646642320=74, srv894052468=91, srv1530652214=17, srv2037591242=46, srv875789698=86, srv2143561448=53, srv4244085=62, srv1906651700=40, srv1571296806=24, srv1241572002=8, srv760357794=78, srv30249479=57, srv1544940064=22, srv1532465721=19, srv899704089=93, srv1706463030=33, srv435281394=64, srv839295578=83, srv794143127=80, srv1539721645=20, srv1209025083=7, srv1174091028=5, srv1632933080=28, srv1320904893=12, srv83647784=82, srv912371304=94, srv812019433=81, srv89479825=92, srv1485405387=16, srv772006989=79, srv1799781224=36, srv1084362363=1, srv340041161=60, srv244699492=54, srv2036954215=45, srv449496579=66, srv2011221269=44, srv977225643=97, srv278240069=56, srv1531237906=18, srv698336798=76, srv1326987750=13, srv2140068891=52, srv1086547822=2, srv1395040171=14, srv27649337=55, srv2071828250=47, srv1187552366=6, srv1547865263=23, srv1604353144=26, srv1597247871=25, srv2129601970=50, srv592444309=71, srv997045154=99, srv703466415=77, srv1092745480=3, srv337567916=59, srv1976545914=43, srv1457341107=15, srv1714639493=34, srv893493858=90, srv1688730175=31, srv1704833961=32, srv1247015677=10, srv1925442669=42, srv1675030529=30, srv2086355439=48, srv859643511=85, srv888752425=89, srv1540375263=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:29,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:29,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:29,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:29,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:29,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:29,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-20 03:53:29,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-20 03:53:29,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-20 03:53:29,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-20 03:53:29,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-20 03:53:29,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-20 03:53:29,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-20 03:53:29,539 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:29,540 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv337567916:83 , srv661318165:83 , srv83647784:83 , srv875789698:83 , srv894052468:83 , srv1084362363:84 , srv1086547822:84 , srv1092745480:84 , srv1132303614:84 , srv1174091028:84 , srv1187552366:84 , srv1209025083:84 , srv1241572002:84 , srv1242014112:84 , srv1247015677:84 , srv125634843:84 , srv1320904893:84 , srv1326987750:84 , srv1395040171:84 , srv1457341107:84 , srv1485405387:84 , srv1530652214:84 , srv1531237906:84 , srv1532465721:84 , srv1540375263:84 , srv1544940064:84 , srv1547865263:84 , srv1571296806:84 , srv1597247871:84 , srv1604353144:84 , srv1611649607:84 , srv1632933080:84 , srv1650790998:84 , srv1675030529:84 , srv1688730175:84 , srv1704833961:84 , srv1706463030:84 , srv1714639493:84 , srv1780669202:84 , srv1799781224:84 , srv1833125715:84 , srv1834781165:84 , srv1846109764:84 , srv1914371605:84 , srv1925442669:84 , srv1976545914:84 , srv2011221269:84 , srv2036954215:84 , srv2037591242:84 , srv2071828250:84 , srv2086355439:84 , srv2112218724:84 , srv2129601970:84 , srv2134264019:84 , srv2140068891:84 , srv2143561448:84 , srv244699492:84 , srv27649337:84 , srv278240069:84 , srv30249479:84 , srv331232902:84 , srv340041161:84 , srv385018931:84 , srv4244085:84 , srv432961836:84 , srv441611867:84 , srv449496579:84 , srv514350414:84 , srv515009166:84 , srv51524327:84 , srv592444309:84 , srv618175898:84 , srv639090973:84 , srv646642320:84 , srv698336798:84 , srv703466415:84 , srv760357794:84 , srv772006989:84 , srv794143127:84 , srv812019433:84 , srv839295578:84 , srv848547324:84 , srv859643511:84 , srv887492430:84 , srv888752425:84 , srv893493858:84 , srv89479825:84 , srv899704089:84 , srv912371304:84 , srv930061792:84 , srv958614823:84 , srv977225643:84 , srv980100006:84 , srv997045154:84 , srv1005514051:85 , srv1906651700:85 , srv435281394:85 , srv472314018:85 , srv888514130:85 , srv1539721645:684 }
2023-05-20 03:53:29,570 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=422 (was 389) - SystemLoadAverage LEAK? -, ProcessCount=176 (was 176), AvailableMemoryMB=7712 (was 8354)
2023-05-20 03:53:29,578 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=422, ProcessCount=176, AvailableMemoryMB=7711
2023-05-20 03:53:29,579 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-20 03:53:29,579 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-20 03:53:29,592 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1043461411:84 , srv1061857976:84 , srv116844193:84 , srv1185744229:84 , srv1364308953:84 , srv1380096267:84 , srv1436458204:84 , srv1447947200:84 , srv146067148:84 , srv1461525026:84 , srv1481437766:84 , srv1515566451:84 , srv1545847114:84 , srv1711528646:84 , srv1726105431:84 , srv1886989673:84 , srv197314469:84 , srv1994359743:84 , srv232983872:84 , srv391444671:84 , srv474766741:84 , srv491909348:84 , srv535406931:84 , srv541856838:84 , srv587851307:84 , srv59521226:84 , srv610533471:84 , srv634427134:84 , srv996284347:84 , srv628017122:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-20 03:53:29,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:53:29,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1711528646=13, srv1364308953=4, srv116844193=2, srv1447947200=7, srv391444671=19, srv1515566451=11, srv1994359743=17, srv1886989673=15, srv1545847114=12, srv197314469=16, srv610533471=26, srv634427134=28, srv474766741=20, srv146067148=8, srv1461525026=9, srv1061857976=1, srv996284347=29, srv587851307=24, srv1185744229=3, srv1380096267=5, srv541856838=23, srv1043461411=0, srv1726105431=14, srv491909348=21, srv535406931=22, srv1436458204=6, srv232983872=18, srv59521226=25, srv628017122=27, srv1481437766=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:53:29,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:53:29,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:53:29,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:53:29,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:53:29,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:53:29,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-20 03:53:29,625 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0940086228984696, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:53:29,625 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991248926136537, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0940086228984696, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-20 03:55:29,613 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6044812 different iterations.  Found a solution that moves 1858 regions; Going from a computed imbalance of 0.991248926136537 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:55:29,622 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1043461411:90 , srv1061857976:90 , srv116844193:90 , srv1185744229:90 , srv1364308953:90 , srv1380096267:90 , srv1436458204:90 , srv1447947200:90 , srv146067148:90 , srv1461525026:90 , srv1481437766:90 , srv1515566451:90 , srv1545847114:90 , srv1711528646:90 , srv1726105431:90 , srv1886989673:90 , srv197314469:90 , srv1994359743:90 , srv232983872:90 , srv391444671:90 , srv474766741:90 , srv491909348:90 , srv535406931:90 , srv541856838:90 , srv587851307:90 , srv59521226:90 , srv610533471:90 , srv628017122:90 , srv634427134:90 , srv996284347:90 }
2023-05-20 03:55:29,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-20 03:55:29,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1711528646=13, srv1364308953=4, srv116844193=2, srv1447947200=7, srv391444671=19, srv1515566451=11, srv1994359743=17, srv1886989673=15, srv1545847114=12, srv197314469=16, srv610533471=26, srv634427134=28, srv474766741=20, srv146067148=8, srv1461525026=9, srv1061857976=1, srv996284347=29, srv587851307=24, srv1185744229=3, srv1380096267=5, srv541856838=23, srv1043461411=0, srv1726105431=14, srv491909348=21, srv535406931=22, srv1436458204=6, srv232983872=18, srv59521226=25, srv628017122=27, srv1481437766=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-20 03:55:29,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-20 03:55:29,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-20 03:55:29,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-20 03:55:29,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-20 03:55:29,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-20 03:55:29,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-20 03:55:29,649 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-20 03:55:29,650 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1043461411:90 , srv1061857976:90 , srv116844193:90 , srv1185744229:90 , srv1364308953:90 , srv1380096267:90 , srv1436458204:90 , srv1447947200:90 , srv146067148:90 , srv1461525026:90 , srv1481437766:90 , srv1515566451:90 , srv1545847114:90 , srv1711528646:90 , srv1726105431:90 , srv1886989673:90 , srv197314469:90 , srv1994359743:90 , srv232983872:90 , srv391444671:90 , srv474766741:90 , srv491909348:90 , srv535406931:90 , srv541856838:90 , srv587851307:90 , srv59521226:90 , srv610533471:90 , srv628017122:90 , srv634427134:90 , srv996284347:90 }
2023-05-20 03:55:29,668 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=237 (was 422), ProcessCount=176 (was 176), AvailableMemoryMB=7727 (was 7711) - AvailableMemoryMB LEAK? -