2023-08-16 03:53:04,922 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-08-16 03:53:05,161 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-08-16 03:53:05,257 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-08-16 03:53:05,312 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-16 03:53:05,318 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-16 03:53:05,330 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=428, ProcessCount=175, AvailableMemoryMB=3297
2023-08-16 03:53:05,331 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-16 03:53:05,332 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-16 03:53:05,463 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003197979:84 , srv1033612130:84 , srv1057118715:84 , srv1067554607:84 , srv1074444813:84 , srv1081108638:84 , srv1091152399:84 , srv1105370373:84 , srv115863221:84 , srv1178796456:84 , srv1192578749:84 , srv1236138235:84 , srv1241832482:84 , srv1244628063:84 , srv1274870320:84 , srv1339462011:84 , srv136809365:84 , srv137847061:84 , srv1386758350:84 , srv1395381446:84 , srv1454486146:84 , srv1475540976:84 , srv1531588347:84 , srv1535877876:84 , srv1538479849:84 , srv1544001605:84 , srv1592446209:84 , srv1606851366:84 , srv1630472286:84 , srv1638747621:84 , srv1663120822:84 , srv1665976209:84 , srv1670776235:84 , srv1687440336:84 , srv1698489303:84 , srv1714261572:84 , srv1714637471:84 , srv1718339325:84 , srv1736689762:84 , srv173722539:84 , srv1742939221:84 , srv1743160527:84 , srv176824456:84 , srv1771338221:84 , srv1777650664:84 , srv1778374801:84 , srv1780951636:84 , srv1785238979:84 , srv1850816242:84 , srv1863029867:84 , srv1892459749:84 , srv1897318865:84 , srv1904115816:84 , srv1937666567:84 , srv1975763555:84 , srv202734202:84 , srv2032523379:84 , srv2082593921:84 , srv2088357609:84 , srv2094133914:84 , srv2096484856:84 , srv209701817:84 , srv2103436097:84 , srv2132353619:84 , srv270479541:84 , srv288286190:84 , srv307243105:84 , srv315284648:84 , srv317791992:84 , srv333064026:84 , srv364609700:84 , srv373964573:84 , srv414879597:84 , srv459722756:84 , srv465540211:84 , srv531670458:84 , srv533314012:84 , srv570558282:84 , srv575202778:84 , srv579445085:84 , srv595840096:84 , srv626262120:84 , srv670506993:84 , srv689865359:84 , srv702741488:84 , srv704666784:84 , srv731335718:84 , srv755223058:84 , srv773416583:84 , srv799232348:84 , srv803655804:84 , srv808092368:84 , srv809013641:84 , srv841308470:84 , srv950617546:84 , srv954173625:84 , srv957098941:84 , srv966055448:84 , srv970862950:84 , srv387542783:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-08-16 03:53:05,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:05,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:05,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:05,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:05,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:05,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:05,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:05,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:05,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:05,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:05,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:05,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:05,691 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:05,691 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-16 03:53:05,692 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-16 03:53:06,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 517 ms to try 5000 different iterations.  Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1765157670079756. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05447226664913533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23641666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,120 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1743160527:83 , srv1897318865:83 , srv202734202:83 , srv626262120:83 , srv1091152399:84 , srv1274870320:84 , srv1531588347:84 , srv1535877876:84 , srv1592446209:84 , srv1778374801:84 , srv1785238979:84 , srv1863029867:84 , srv1892459749:84 , srv209701817:84 , srv307243105:84 , srv373964573:84 , srv414879597:84 , srv533314012:84 , srv755223058:84 , srv809013641:84 , srv1067554607:85 , srv1244628063:85 , srv1475540976:85 , srv1665976209:85 , srv1718339325:85 , srv173722539:85 , srv2132353619:85 , srv315284648:85 , srv364609700:85 , srv459722756:85 , srv465540211:85 , srv575202778:85 , srv595840096:85 , srv670506993:85 , srv799232348:85 , srv950617546:85 , srv1003197979:86 , srv1033612130:86 , srv1057118715:86 , srv1074444813:86 , srv1538479849:86 , srv1606851366:86 , srv1663120822:86 , srv1687440336:86 , srv1714261572:86 , srv176824456:86 , srv1975763555:86 , srv2032523379:86 , srv2096484856:86 , srv2103436097:86 , srv689865359:86 , srv803655804:86 , srv841308470:86 , srv957098941:86 , srv970862950:86 , srv1105370373:87 , srv1178796456:87 , srv1192578749:87 , srv1236138235:87 , srv1339462011:87 , srv1395381446:87 , srv1777650664:87 , srv1780951636:87 , srv1850816242:87 , srv1937666567:87 , srv2082593921:87 , srv2094133914:87 , srv270479541:87 , srv288286190:87 , srv317791992:87 , srv333064026:87 , srv531670458:87 , srv570558282:87 , srv773416583:87 , srv966055448:87 , srv1081108638:88 , srv1386758350:88 , srv1544001605:88 , srv1698489303:88 , srv2088357609:88 , srv579445085:88 , srv731335718:88 , srv954173625:88 , srv1241832482:89 , srv137847061:89 , srv704666784:89 , srv136809365:90 , srv1638747621:90 , srv1742939221:90 , srv1771338221:90 , srv808092368:90 , srv1630472286:91 , srv1670776235:91 , srv1904115816:91 , srv702741488:91 , srv1736689762:93 , srv1454486146:95 , srv1714637471:96 , srv115863221:97 , srv387542783:424 }
2023-08-16 03:53:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:06,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:06,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:06,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:06,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:06,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:06,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:06,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:06,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:06,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:06,309 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05447226664913533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23641666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,309 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-16 03:53:06,309 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17648284990242422, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037600097109664374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05447226664913533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037600097109664374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23641666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-16 03:53:06,501 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 314 ms to try 5000 different iterations.  Found a solution that moves 2569 regions; Going from a computed imbalance of 0.17648284990242422 to a new imbalance of 0.01428630307295524. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022709582839898); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035463897008384794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022709582839898); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,511 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv115863221:75 , srv1714637471:75 , srv1454486146:77 , srv1736689762:79 , srv1630472286:80 , srv1670776235:80 , srv1904115816:80 , srv137847061:81 , srv136809365:82 , srv702741488:82 , srv808092368:82 , srv1386758350:83 , srv1544001605:83 , srv1638747621:83 , srv1742939221:83 , srv1771338221:83 , srv1081108638:84 , srv1236138235:84 , srv1339462011:84 , srv1395381446:84 , srv1698489303:84 , srv1714261572:84 , srv176824456:84 , srv1850816242:84 , srv1937666567:84 , srv2032523379:84 , srv2088357609:84 , srv2094133914:84 , srv317791992:84 , srv333064026:84 , srv579445085:84 , srv731335718:84 , srv1057118715:85 , srv1178796456:85 , srv1241832482:85 , srv1244628063:85 , srv1475540976:85 , srv1687440336:85 , srv1777650664:85 , srv1780951636:85 , srv2082593921:85 , srv270479541:85 , srv288286190:85 , srv315284648:85 , srv531670458:85 , srv570558282:85 , srv704666784:85 , srv773416583:85 , srv966055448:85 , srv970862950:85 , srv1033612130:86 , srv1192578749:86 , srv1538479849:86 , srv1718339325:86 , srv1975763555:86 , srv2096484856:86 , srv2103436097:86 , srv2132353619:86 , srv364609700:86 , srv465540211:86 , srv689865359:86 , srv799232348:86 , srv809013641:86 , srv1003197979:87 , srv1091152399:87 , srv1105370373:87 , srv1531588347:87 , srv1535877876:87 , srv1606851366:87 , srv1663120822:87 , srv173722539:87 , srv1778374801:87 , srv209701817:87 , srv459722756:87 , srv533314012:87 , srv803655804:87 , srv841308470:87 , srv950617546:87 , srv954173625:87 , srv957098941:87 , srv1074444813:88 , srv1592446209:88 , srv1785238979:88 , srv1863029867:88 , srv1892459749:88 , srv373964573:88 , srv595840096:88 , srv670506993:88 , srv755223058:88 , srv1665976209:89 , srv1897318865:89 , srv202734202:89 , srv414879597:89 , srv575202778:89 , srv626262120:89 , srv307243105:90 , srv1067554607:91 , srv1274870320:91 , srv1743160527:91 , srv387542783:552 }
2023-08-16 03:53:06,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:06,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:06,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:06,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:06,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:06,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:06,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:06,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:06,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:06,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:06,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:06,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:06,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022709582839898); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035463897008384794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022709582839898); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,635 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-16 03:53:06,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01426831455047884, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022709582839898); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035463897008384794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022709582839898); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-16 03:53:06,792 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 232 ms to try 5000 different iterations.  Found a solution that moves 931 regions; Going from a computed imbalance of 0.01426831455047884 to a new imbalance of 6.417899093429761E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007459413397542786); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009813067629253163); MoveCostFunction : (multiplier=7.0, imbalance=0.10344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007459413397542786); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,794 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1274870320:81 , srv1067554607:82 , srv1241832482:82 , srv954173625:82 , srv1105370373:83 , srv1743160527:83 , srv704666784:83 , srv1074444813:84 , srv1665976209:84 , srv2088357609:84 , srv2103436097:84 , srv307243105:84 , srv531670458:84 , srv626262120:84 , srv670506993:84 , srv689865359:84 , srv702741488:84 , srv755223058:84 , srv773416583:84 , srv1003197979:85 , srv1033612130:85 , srv1081108638:85 , srv115863221:85 , srv1178796456:85 , srv1192578749:85 , srv1339462011:85 , srv136809365:85 , srv1454486146:85 , srv1531588347:85 , srv1538479849:85 , srv1606851366:85 , srv1638747621:85 , srv1663120822:85 , srv1698489303:85 , srv1736689762:85 , srv173722539:85 , srv1742939221:85 , srv1771338221:85 , srv1777650664:85 , srv1785238979:85 , srv1892459749:85 , srv2096484856:85 , srv2132353619:85 , srv288286190:85 , srv373964573:85 , srv414879597:85 , srv533314012:85 , srv570558282:85 , srv575202778:85 , srv579445085:85 , srv595840096:85 , srv731335718:85 , srv803655804:85 , srv841308470:85 , srv950617546:85 , srv957098941:85 , srv966055448:85 , srv1057118715:86 , srv1091152399:86 , srv1592446209:86 , srv1630472286:86 , srv1670776235:86 , srv1780951636:86 , srv1850816242:86 , srv1863029867:86 , srv1897318865:86 , srv1975763555:86 , srv202734202:86 , srv2082593921:86 , srv209701817:86 , srv315284648:86 , srv317791992:86 , srv364609700:86 , srv459722756:86 , srv465540211:86 , srv799232348:86 , srv808092368:86 , srv970862950:86 , srv1236138235:87 , srv137847061:87 , srv1386758350:87 , srv1395381446:87 , srv1475540976:87 , srv1535877876:87 , srv1544001605:87 , srv1687440336:87 , srv1714637471:87 , srv1718339325:87 , srv176824456:87 , srv1904115816:87 , srv1937666567:87 , srv2032523379:87 , srv2094133914:87 , srv270479541:87 , srv809013641:87 , srv1244628063:88 , srv1714261572:88 , srv1778374801:88 , srv333064026:89 , srv387542783:548 }
2023-08-16 03:53:06,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:06,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:06,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:06,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:06,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:06,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:06,893 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007459413397542786); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009813067629253163); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007459413397542786); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:06,893 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-16 03:53:06,893 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.352709080095667E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007459413397542786); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009813067629253163); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007459413397542786); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-16 03:53:07,105 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 283 ms to try 5000 different iterations.  Found a solution that moves 271 regions; Going from a computed imbalance of 6.352709080095667E-4 to a new imbalance of 3.658296135443865E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.030111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:07,106 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv333064026:82 , srv1778374801:83 , srv270479541:83 , srv307243105:83 , srv1003197979:84 , srv1067554607:84 , srv1074444813:84 , srv1192578749:84 , srv1244628063:84 , srv137847061:84 , srv1386758350:84 , srv1395381446:84 , srv1535877876:84 , srv1544001605:84 , srv1592446209:84 , srv1606851366:84 , srv1630472286:84 , srv1638747621:84 , srv1670776235:84 , srv1687440336:84 , srv1714261572:84 , srv1718339325:84 , srv1742939221:84 , srv1771338221:84 , srv1777650664:84 , srv1780951636:84 , srv1785238979:84 , srv1863029867:84 , srv1897318865:84 , srv1904115816:84 , srv1937666567:84 , srv1975763555:84 , srv202734202:84 , srv2082593921:84 , srv2094133914:84 , srv373964573:84 , srv414879597:84 , srv459722756:84 , srv465540211:84 , srv575202778:84 , srv595840096:84 , srv803655804:84 , srv841308470:84 , srv970862950:84 , srv1033612130:85 , srv1057118715:85 , srv1081108638:85 , srv1105370373:85 , srv115863221:85 , srv1236138235:85 , srv136809365:85 , srv1454486146:85 , srv1475540976:85 , srv1531588347:85 , srv1538479849:85 , srv1663120822:85 , srv1665976209:85 , srv1698489303:85 , srv1714637471:85 , srv1736689762:85 , srv173722539:85 , srv1743160527:85 , srv176824456:85 , srv1850816242:85 , srv1892459749:85 , srv2032523379:85 , srv2096484856:85 , srv288286190:85 , srv317791992:85 , srv364609700:85 , srv570558282:85 , srv579445085:85 , srv670506993:85 , srv702741488:85 , srv704666784:85 , srv731335718:85 , srv799232348:85 , srv808092368:85 , srv809013641:85 , srv950617546:85 , srv954173625:85 , srv957098941:85 , srv966055448:85 , srv1091152399:86 , srv1178796456:86 , srv1241832482:86 , srv1274870320:86 , srv1339462011:86 , srv2088357609:86 , srv209701817:86 , srv2103436097:86 , srv2132353619:86 , srv315284648:86 , srv531670458:86 , srv533314012:86 , srv626262120:86 , srv689865359:86 , srv755223058:86 , srv773416583:86 , srv387542783:618 }
2023-08-16 03:53:07,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:07,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:07,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:07,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:07,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:07,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:07,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:07,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:07,211 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:07,211 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-16 03:53:07,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.4685378796593394E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-16 03:53:07,353 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 220 ms to try 5000 different iterations.  Found a solution that moves 34 regions; Going from a computed imbalance of 3.4685378796593394E-5 to a new imbalance of 7.311312384767599E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:07,354 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1091152399:83 , srv1178796456:83 , srv1236138235:83 , srv1663120822:83 , srv1665976209:83 , srv1714637471:83 , srv209701817:83 , srv575202778:83 , srv808092368:83 , srv957098941:83 , srv1003197979:84 , srv1033612130:84 , srv1057118715:84 , srv1067554607:84 , srv1074444813:84 , srv1081108638:84 , srv1105370373:84 , srv115863221:84 , srv1192578749:84 , srv1241832482:84 , srv1244628063:84 , srv1274870320:84 , srv1339462011:84 , srv136809365:84 , srv1386758350:84 , srv1395381446:84 , srv1454486146:84 , srv1475540976:84 , srv1535877876:84 , srv1538479849:84 , srv1544001605:84 , srv1592446209:84 , srv1606851366:84 , srv1638747621:84 , srv1687440336:84 , srv1698489303:84 , srv1714261572:84 , srv1718339325:84 , srv1736689762:84 , srv173722539:84 , srv1742939221:84 , srv1743160527:84 , srv176824456:84 , srv1771338221:84 , srv1778374801:84 , srv1780951636:84 , srv1850816242:84 , srv1863029867:84 , srv1892459749:84 , srv1897318865:84 , srv1904115816:84 , srv1937666567:84 , srv1975763555:84 , srv202734202:84 , srv2032523379:84 , srv2082593921:84 , srv2088357609:84 , srv2094133914:84 , srv2096484856:84 , srv2103436097:84 , srv2132353619:84 , srv270479541:84 , srv288286190:84 , srv315284648:84 , srv317791992:84 , srv333064026:84 , srv364609700:84 , srv414879597:84 , srv459722756:84 , srv531670458:84 , srv533314012:84 , srv570558282:84 , srv579445085:84 , srv595840096:84 , srv626262120:84 , srv670506993:84 , srv689865359:84 , srv702741488:84 , srv704666784:84 , srv731335718:84 , srv755223058:84 , srv773416583:84 , srv799232348:84 , srv803655804:84 , srv809013641:84 , srv841308470:84 , srv950617546:84 , srv954173625:84 , srv966055448:84 , srv137847061:85 , srv1531588347:85 , srv1630472286:85 , srv1670776235:85 , srv1777650664:85 , srv1785238979:85 , srv307243105:85 , srv373964573:85 , srv465540211:85 , srv970862950:85 , srv387542783:684 }
2023-08-16 03:53:07,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:07,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1544001605=25, srv1244628063=13, srv1687440336=33, srv704666784=86, srv373964573=71, srv1718339325=37, srv773416583=89, srv1670776235=32, srv1698489303=34, srv1454486146=20, srv2103436097=62, srv307243105=66, srv1192578749=10, srv1033612130=1, srv317791992=68, srv176824456=42, srv1743160527=41, srv579445085=80, srv1395381446=19, srv1531588347=22, srv689865359=84, srv459722756=74, srv1975763555=54, srv531670458=76, srv1057118715=2, srv808092368=92, srv533314012=77, srv414879597=73, srv1386758350=18, srv2096484856=60, srv702741488=85, srv1538479849=24, srv1638747621=29, srv1771338221=43, srv799232348=90, srv954173625=96, srv2032523379=56, srv1241832482=12, srv1339462011=15, srv575202778=79, srv755223058=88, srv1178796456=9, srv315284648=67, srv1780951636=46, srv1778374801=45, srv2132353619=63, srv966055448=98, srv1937666567=53, srv1736689762=38, srv2094133914=59, srv1074444813=4, srv1742939221=40, srv1105370373=7, srv288286190=65, srv1592446209=26, srv270479541=64, srv333064026=69, srv1904115816=52, srv841308470=94, srv1892459749=50, srv209701817=61, srv731335718=87, srv1091152399=6, srv595840096=81, srv809013641=93, srv465540211=75, srv1067554607=3, srv1003197979=0, srv173722539=39, srv1785238979=47, srv202734202=55, srv1236138235=11, srv626262120=82, srv1475540976=21, srv970862950=99, srv803655804=91, srv137847061=17, srv387542783=72, srv2088357609=58, srv2082593921=57, srv1897318865=51, srv1665976209=31, srv670506993=83, srv136809365=16, srv570558282=78, srv1850816242=48, srv1777650664=44, srv1714637471=36, srv1081108638=5, srv115863221=8, srv1714261572=35, srv1274870320=14, srv1535877876=23, srv950617546=95, srv1663120822=30, srv1863029867=49, srv1606851366=27, srv1630472286=28, srv364609700=70, srv957098941=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:07,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:07,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:07,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-16 03:53:07,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-16 03:53:07,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-16 03:53:07,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-16 03:53:07,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-16 03:53:07,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-16 03:53:07,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-16 03:53:07,453 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:07,453 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1091152399:83 , srv1178796456:83 , srv1236138235:83 , srv1663120822:83 , srv1665976209:83 , srv1714637471:83 , srv209701817:83 , srv575202778:83 , srv808092368:83 , srv957098941:83 , srv1003197979:84 , srv1033612130:84 , srv1057118715:84 , srv1067554607:84 , srv1074444813:84 , srv1081108638:84 , srv1105370373:84 , srv115863221:84 , srv1192578749:84 , srv1241832482:84 , srv1244628063:84 , srv1274870320:84 , srv1339462011:84 , srv136809365:84 , srv1386758350:84 , srv1395381446:84 , srv1454486146:84 , srv1475540976:84 , srv1535877876:84 , srv1538479849:84 , srv1544001605:84 , srv1592446209:84 , srv1606851366:84 , srv1638747621:84 , srv1687440336:84 , srv1698489303:84 , srv1714261572:84 , srv1718339325:84 , srv1736689762:84 , srv173722539:84 , srv1742939221:84 , srv1743160527:84 , srv176824456:84 , srv1771338221:84 , srv1778374801:84 , srv1780951636:84 , srv1850816242:84 , srv1863029867:84 , srv1892459749:84 , srv1897318865:84 , srv1904115816:84 , srv1937666567:84 , srv1975763555:84 , srv202734202:84 , srv2032523379:84 , srv2082593921:84 , srv2088357609:84 , srv2094133914:84 , srv2096484856:84 , srv2103436097:84 , srv2132353619:84 , srv270479541:84 , srv288286190:84 , srv315284648:84 , srv317791992:84 , srv333064026:84 , srv364609700:84 , srv414879597:84 , srv459722756:84 , srv531670458:84 , srv533314012:84 , srv570558282:84 , srv579445085:84 , srv595840096:84 , srv626262120:84 , srv670506993:84 , srv689865359:84 , srv702741488:84 , srv704666784:84 , srv731335718:84 , srv755223058:84 , srv773416583:84 , srv799232348:84 , srv803655804:84 , srv809013641:84 , srv841308470:84 , srv950617546:84 , srv954173625:84 , srv966055448:84 , srv137847061:85 , srv1531588347:85 , srv1630472286:85 , srv1670776235:85 , srv1777650664:85 , srv1785238979:85 , srv307243105:85 , srv373964573:85 , srv465540211:85 , srv970862950:85 , srv387542783:684 }
2023-08-16 03:53:07,479 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 428), ProcessCount=175 (was 175), AvailableMemoryMB=2807 (was 3297)
2023-08-16 03:53:07,486 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=175, AvailableMemoryMB=2806
2023-08-16 03:53:07,487 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-16 03:53:07,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-08-16 03:53:07,496 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1062861342:84 , srv1143590829:84 , srv1148607504:84 , srv1257495527:84 , srv128372552:84 , srv1305770459:84 , srv135285603:84 , srv1371966528:84 , srv1484341287:84 , srv15345289:84 , srv15841937:84 , srv1604187820:84 , srv1627688692:84 , srv1634090027:84 , srv1734827789:84 , srv1766329950:84 , srv1783624921:84 , srv1812268387:84 , srv1860044761:84 , srv1883503254:84 , srv1978359090:84 , srv2068210688:84 , srv214747173:84 , srv394285016:84 , srv534361384:84 , srv60736766:84 , srv619364511:84 , srv663957735:84 , srv72678054:84 , srv182333093:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-08-16 03:53:07,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:53:07,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1484341287=8, srv1143590829=1, srv1883503254=20, srv1305770459=5, srv1371966528=7, srv1604187820=11, srv182333093=18, srv1783624921=16, srv214747173=23, srv1627688692=12, srv15841937=10, srv1766329950=15, srv534361384=25, srv1812268387=17, srv15345289=9, srv1978359090=21, srv60736766=26, srv619364511=27, srv1634090027=13, srv1062861342=0, srv1860044761=19, srv663957735=28, srv1148607504=2, srv72678054=29, srv128372552=4, srv394285016=24, srv1734827789=14, srv135285603=6, srv2068210688=22, srv1257495527=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:53:07,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:53:07,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:53:07,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-16 03:53:07,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.094891675142368, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:53:07,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912492043834156, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.094891675142368, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-08-16 03:55:07,510 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 10738024 different iterations.  Found a solution that moves 1832 regions; Going from a computed imbalance of 0.9912492043834156 to a new imbalance of 4.275979392340115E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6785185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:55:07,517 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1062861342:90 , srv1143590829:90 , srv1148607504:90 , srv1257495527:90 , srv128372552:90 , srv1305770459:90 , srv135285603:90 , srv1371966528:90 , srv1484341287:90 , srv15345289:90 , srv15841937:90 , srv1604187820:90 , srv1627688692:90 , srv1634090027:90 , srv1734827789:90 , srv1766329950:90 , srv1783624921:90 , srv1812268387:90 , srv182333093:90 , srv1860044761:90 , srv1883503254:90 , srv1978359090:90 , srv2068210688:90 , srv214747173:90 , srv394285016:90 , srv534361384:90 , srv60736766:90 , srv619364511:90 , srv663957735:90 , srv72678054:90 }
2023-08-16 03:55:07,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-16 03:55:07,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1484341287=8, srv1143590829=1, srv1883503254=20, srv1305770459=5, srv1371966528=7, srv1604187820=11, srv182333093=18, srv1783624921=16, srv214747173=23, srv1627688692=12, srv15841937=10, srv1766329950=15, srv534361384=25, srv1812268387=17, srv15345289=9, srv1978359090=21, srv60736766=26, srv619364511=27, srv1634090027=13, srv1062861342=0, srv1860044761=19, srv663957735=28, srv1148607504=2, srv72678054=29, srv128372552=4, srv394285016=24, srv1734827789=14, srv135285603=6, srv2068210688=22, srv1257495527=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-16 03:55:07,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-16 03:55:07,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-16 03:55:07,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-16 03:55:07,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-16 03:55:07,531 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-16 03:55:07,531 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1062861342:90 , srv1143590829:90 , srv1148607504:90 , srv1257495527:90 , srv128372552:90 , srv1305770459:90 , srv135285603:90 , srv1371966528:90 , srv1484341287:90 , srv15345289:90 , srv15841937:90 , srv1604187820:90 , srv1627688692:90 , srv1634090027:90 , srv1734827789:90 , srv1766329950:90 , srv1783624921:90 , srv1812268387:90 , srv182333093:90 , srv1860044761:90 , srv1883503254:90 , srv1978359090:90 , srv2068210688:90 , srv214747173:90 , srv394285016:90 , srv534361384:90 , srv60736766:90 , srv619364511:90 , srv663957735:90 , srv72678054:90 }
2023-08-16 03:55:07,542 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=141 (was 409), ProcessCount=174 (was 175), AvailableMemoryMB=2612 (was 2806)