2024-03-11 05:54:21,073 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-11 05:54:21,318 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-11 05:54:21,408 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-11 05:54:21,462 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-11 05:54:21,469 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-11 05:54:21,481 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=245, ProcessCount=174, AvailableMemoryMB=12914
2024-03-11 05:54:21,483 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-11 05:54:21,483 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-11 05:54:21,591 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023700642:84 , srv102894050:84 , srv106397765:84 , srv1077731035:84 , srv109880554:84 , srv1119789171:84 , srv1218629041:84 , srv1245570927:84 , srv1300824360:84 , srv1303617665:84 , srv1308984770:84 , srv1322442414:84 , srv1327982819:84 , srv1340284044:84 , srv1352546939:84 , srv1367221697:84 , srv1371749680:84 , srv1447066897:84 , srv1486841855:84 , srv1501159122:84 , srv1517469870:84 , srv1526000588:84 , srv156604371:84 , srv1582573339:84 , srv1604970999:84 , srv1637775857:84 , srv1640059902:84 , srv165822613:84 , srv1660908087:84 , srv1678237254:84 , srv1692048201:84 , srv1741894218:84 , srv1743271576:84 , srv1768027682:84 , srv1798785208:84 , srv1800139190:84 , srv1824744940:84 , srv1827416250:84 , srv1846629290:84 , srv1847757881:84 , srv1862566385:84 , srv187937683:84 , srv1926466757:84 , srv1989506446:84 , srv2002406083:84 , srv2003894452:84 , srv2007013775:84 , srv2007326313:84 , srv2025878371:84 , srv2032237776:84 , srv2047248170:84 , srv2053273618:84 , srv2070006166:84 , srv208248494:84 , srv2089357100:84 , srv2104469115:84 , srv2123255684:84 , srv2138504829:84 , srv2145338961:84 , srv239003820:84 , srv257492856:84 , srv262179332:84 , srv272123373:84 , srv285702461:84 , srv304352780:84 , srv318300952:84 , srv351466036:84 , srv360458112:84 , srv421532963:84 , srv448188414:84 , srv50244372:84 , srv508406925:84 , srv550588547:84 , srv556731248:84 , srv593879846:84 , srv595261913:84 , srv604843598:84 , srv662282987:84 , srv674338024:84 , srv685129980:84 , srv688836406:84 , srv712878557:84 , srv783640872:84 , srv816801523:84 , srv826798251:84 , srv848439245:84 , srv853699466:84 , srv862008242:84 , srv871306257:84 , srv883937317:84 , srv907683209:84 , srv924324694:84 , srv926088466:84 , srv934275245:84 , srv940415693:84 , srv94390347:84 , srv954237402:84 , srv976513261:84 , srv977231113:84 , srv1027044161:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-11 05:54:21,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:21,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv106397765=3, srv109880554=5, srv1637775857=26, srv1245570927=8, srv1077731035=4, srv2007013775=47, srv595261913=76, srv1300824360=9, srv2123255684=57, srv783640872=83, srv871306257=89, srv1989506446=44, srv318300952=66, srv1678237254=30, srv1119789171=6, srv50244372=71, srv448188414=70, srv1303617665=10, srv421532963=69, srv2138504829=58, srv2145338961=59, srv272123373=63, srv883937317=90, srv940415693=95, srv1517469870=21, srv862008242=88, srv257492856=61, srv1027044161=1, srv662282987=78, srv593879846=75, srv685129980=80, srv826798251=85, srv351466036=67, srv1322442414=12, srv1743271576=33, srv1926466757=43, srv604843598=77, srv508406925=72, srv1827416250=38, srv304352780=65, srv1582573339=24, srv1604970999=25, srv2053273618=52, srv976513261=98, srv1847757881=40, srv1340284044=14, srv2089357100=55, srv187937683=42, srv285702461=64, srv926088466=93, srv1486841855=19, srv1768027682=34, srv1741894218=32, srv1308984770=11, srv1447066897=18, srv1023700642=0, srv2047248170=51, srv2032237776=50, srv2070006166=53, srv1218629041=7, srv816801523=84, srv1501159122=20, srv2002406083=45, srv1327982819=13, srv2007326313=48, srv2003894452=46, srv102894050=2, srv2104469115=56, srv1660908087=29, srv712878557=82, srv934275245=94, srv1526000588=22, srv1798785208=35, srv1367221697=16, srv1352546939=15, srv1371749680=17, srv954237402=97, srv924324694=92, srv1846629290=39, srv360458112=68, srv239003820=60, srv208248494=54, srv848439245=86, srv1640059902=27, srv1824744940=37, srv853699466=87, srv977231113=99, srv550588547=73, srv165822613=28, srv674338024=79, srv1862566385=41, srv556731248=74, srv688836406=81, srv94390347=96, srv262179332=62, srv156604371=23, srv2025878371=49, srv1692048201=31, srv907683209=91, srv1800139190=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:21,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:21,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:21,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:21,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-11 05:54:21,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-11 05:54:21,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-11 05:54:21,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-11 05:54:21,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-11 05:54:21,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-11 05:54:21,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-11 05:54:21,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-11 05:54:21,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-11 05:54:21,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-11 05:54:21,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-11 05:54:21,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-11 05:54:21,929 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:21,929 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-11 05:54:21,930 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-11 05:54:22,205 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 507 ms to try 5000 different iterations.  Found a solution that moves 4668 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1821189073785263. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040252972353165994, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477840268457451, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5186666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040252972353165994, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17741666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,214 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv187937683:80 , srv940415693:80 , srv1077731035:81 , srv1218629041:81 , srv883937317:81 , srv1486841855:82 , srv1741894218:82 , srv2007326313:82 , srv550588547:82 , srv674338024:82 , srv783640872:82 , srv208248494:83 , srv318300952:83 , srv662282987:83 , srv853699466:83 , srv94390347:83 , srv102894050:84 , srv1322442414:84 , srv1352546939:84 , srv1501159122:84 , srv156604371:84 , srv1678237254:84 , srv2032237776:84 , srv2053273618:84 , srv2070006166:84 , srv2138504829:84 , srv239003820:84 , srv304352780:84 , srv508406925:84 , srv816801523:84 , srv977231113:84 , srv1340284044:85 , srv1517469870:85 , srv1640059902:85 , srv1743271576:85 , srv1798785208:85 , srv1989506446:85 , srv2047248170:85 , srv685129980:85 , srv934275245:85 , srv1023700642:86 , srv109880554:86 , srv1119789171:86 , srv1245570927:86 , srv1371749680:86 , srv1447066897:86 , srv1582573339:86 , srv1800139190:86 , srv1827416250:86 , srv448188414:86 , srv556731248:86 , srv593879846:86 , srv595261913:86 , srv826798251:86 , srv1303617665:87 , srv1308984770:87 , srv1768027682:87 , srv1847757881:87 , srv2007013775:87 , srv2123255684:87 , srv2145338961:87 , srv262179332:87 , srv272123373:87 , srv351466036:87 , srv360458112:87 , srv604843598:87 , srv848439245:87 , srv862008242:87 , srv924324694:87 , srv976513261:87 , srv1327982819:88 , srv1367221697:88 , srv1526000588:88 , srv1604970999:88 , srv1660908087:88 , srv1692048201:88 , srv1926466757:88 , srv2003894452:88 , srv2089357100:88 , srv2104469115:88 , srv106397765:89 , srv165822613:89 , srv926088466:89 , srv1824744940:90 , srv1862566385:90 , srv50244372:90 , srv712878557:90 , srv954237402:90 , srv1637775857:91 , srv1846629290:91 , srv257492856:91 , srv285702461:91 , srv688836406:91 , srv2002406083:92 , srv2025878371:92 , srv421532963:92 , srv1300824360:94 , srv871306257:99 , srv907683209:101 , srv1027044161:447 }
2024-03-11 05:54:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv106397765=3, srv109880554=5, srv1637775857=26, srv1245570927=8, srv1077731035=4, srv2007013775=47, srv595261913=76, srv1300824360=9, srv2123255684=57, srv783640872=83, srv871306257=89, srv1989506446=44, srv318300952=66, srv1678237254=30, srv1119789171=6, srv50244372=71, srv448188414=70, srv1303617665=10, srv421532963=69, srv2138504829=58, srv2145338961=59, srv272123373=63, srv883937317=90, srv940415693=95, srv1517469870=21, srv862008242=88, srv257492856=61, srv1027044161=1, srv662282987=78, srv593879846=75, srv685129980=80, srv826798251=85, srv351466036=67, srv1322442414=12, srv1743271576=33, srv1926466757=43, srv604843598=77, srv508406925=72, srv1827416250=38, srv304352780=65, srv1582573339=24, srv1604970999=25, srv2053273618=52, srv976513261=98, srv1847757881=40, srv1340284044=14, srv2089357100=55, srv187937683=42, srv285702461=64, srv926088466=93, srv1486841855=19, srv1768027682=34, srv1741894218=32, srv1308984770=11, srv1447066897=18, srv1023700642=0, srv2047248170=51, srv2032237776=50, srv2070006166=53, srv1218629041=7, srv816801523=84, srv1501159122=20, srv2002406083=45, srv1327982819=13, srv2007326313=48, srv2003894452=46, srv102894050=2, srv2104469115=56, srv1660908087=29, srv712878557=82, srv934275245=94, srv1526000588=22, srv1798785208=35, srv1367221697=16, srv1352546939=15, srv1371749680=17, srv954237402=97, srv924324694=92, srv1846629290=39, srv360458112=68, srv239003820=60, srv208248494=54, srv848439245=86, srv1640059902=27, srv1824744940=37, srv853699466=87, srv977231113=99, srv550588547=73, srv165822613=28, srv674338024=79, srv1862566385=41, srv556731248=74, srv688836406=81, srv94390347=96, srv262179332=62, srv156604371=23, srv2025878371=49, srv1692048201=31, srv907683209=91, srv1800139190=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:22,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:22,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:22,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:22,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:22,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:22,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:22,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:22,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:22,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-11 05:54:22,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-11 05:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-11 05:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-11 05:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-11 05:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-11 05:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-11 05:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-11 05:54:22,364 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040252972353165994, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477840268457451, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040252972353165994, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17741666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,364 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-11 05:54:22,364 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18208622134391367, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040252972353165994, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477840268457451, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040252972353165994, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17741666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-11 05:54:22,538 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 257 ms to try 5000 different iterations.  Found a solution that moves 2613 regions; Going from a computed imbalance of 0.18208622134391367 to a new imbalance of 0.01644072803152171. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0213726489574126); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030440094063575535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29033333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0213726489574126); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,546 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv907683209:70 , srv871306257:74 , srv1300824360:78 , srv2025878371:79 , srv421532963:80 , srv1637775857:81 , srv1824744940:81 , srv1926466757:81 , srv2002406083:81 , srv257492856:81 , srv50244372:81 , srv1846629290:82 , srv285702461:82 , srv712878557:82 , srv954237402:82 , srv1526000588:83 , srv1604970999:83 , srv1660908087:83 , srv1847757881:83 , srv2003894452:83 , srv688836406:83 , srv1367221697:84 , srv165822613:84 , srv2089357100:84 , srv272123373:84 , srv351466036:84 , srv360458112:84 , srv556731248:84 , srv604843598:84 , srv848439245:84 , srv924324694:84 , srv926088466:84 , srv1119789171:85 , srv1245570927:85 , srv1327982819:85 , srv1371749680:85 , srv1447066897:85 , srv1692048201:85 , srv1768027682:85 , srv1800139190:85 , srv1862566385:85 , srv2145338961:85 , srv593879846:85 , srv106397765:86 , srv1303617665:86 , srv1308984770:86 , srv1827416250:86 , srv1989506446:86 , srv2047248170:86 , srv208248494:86 , srv2104469115:86 , srv239003820:86 , srv262179332:86 , srv685129980:86 , srv862008242:86 , srv976513261:86 , srv102894050:87 , srv109880554:87 , srv1340284044:87 , srv1517469870:87 , srv1743271576:87 , srv2007013775:87 , srv2053273618:87 , srv2123255684:87 , srv826798251:87 , srv934275245:87 , srv977231113:87 , srv1322442414:88 , srv1352546939:88 , srv1501159122:88 , srv156604371:88 , srv1582573339:88 , srv1640059902:88 , srv1678237254:88 , srv1741894218:88 , srv2070006166:88 , srv2138504829:88 , srv304352780:88 , srv448188414:88 , srv508406925:88 , srv595261913:88 , srv2032237776:89 , srv662282987:89 , srv674338024:89 , srv783640872:89 , srv816801523:89 , srv1023700642:90 , srv1798785208:90 , srv187937683:90 , srv2007326313:90 , srv318300952:90 , srv550588547:90 , srv853699466:90 , srv94390347:90 , srv1486841855:91 , srv940415693:91 , srv1077731035:92 , srv883937317:92 , srv1218629041:93 , srv1027044161:517 }
2024-03-11 05:54:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv106397765=3, srv109880554=5, srv1637775857=26, srv1245570927=8, srv1077731035=4, srv2007013775=47, srv595261913=76, srv1300824360=9, srv2123255684=57, srv783640872=83, srv871306257=89, srv1989506446=44, srv318300952=66, srv1678237254=30, srv1119789171=6, srv50244372=71, srv448188414=70, srv1303617665=10, srv421532963=69, srv2138504829=58, srv2145338961=59, srv272123373=63, srv883937317=90, srv940415693=95, srv1517469870=21, srv862008242=88, srv257492856=61, srv1027044161=1, srv662282987=78, srv593879846=75, srv685129980=80, srv826798251=85, srv351466036=67, srv1322442414=12, srv1743271576=33, srv1926466757=43, srv604843598=77, srv508406925=72, srv1827416250=38, srv304352780=65, srv1582573339=24, srv1604970999=25, srv2053273618=52, srv976513261=98, srv1847757881=40, srv1340284044=14, srv2089357100=55, srv187937683=42, srv285702461=64, srv926088466=93, srv1486841855=19, srv1768027682=34, srv1741894218=32, srv1308984770=11, srv1447066897=18, srv1023700642=0, srv2047248170=51, srv2032237776=50, srv2070006166=53, srv1218629041=7, srv816801523=84, srv1501159122=20, srv2002406083=45, srv1327982819=13, srv2007326313=48, srv2003894452=46, srv102894050=2, srv2104469115=56, srv1660908087=29, srv712878557=82, srv934275245=94, srv1526000588=22, srv1798785208=35, srv1367221697=16, srv1352546939=15, srv1371749680=17, srv954237402=97, srv924324694=92, srv1846629290=39, srv360458112=68, srv239003820=60, srv208248494=54, srv848439245=86, srv1640059902=27, srv1824744940=37, srv853699466=87, srv977231113=99, srv550588547=73, srv165822613=28, srv674338024=79, srv1862566385=41, srv556731248=74, srv688836406=81, srv94390347=96, srv262179332=62, srv156604371=23, srv2025878371=49, srv1692048201=31, srv907683209=91, srv1800139190=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:22,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:22,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-11 05:54:22,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-11 05:54:22,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-11 05:54:22,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-11 05:54:22,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-11 05:54:22,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-11 05:54:22,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0213726489574126); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030440094063575535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0213726489574126); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,653 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-11 05:54:22,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016422431414460273, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0213726489574126); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030440094063575535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0213726489574126); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-11 05:54:22,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 227 ms to try 5000 different iterations.  Found a solution that moves 1055 regions; Going from a computed imbalance of 0.016422431414460273 to a new imbalance of 6.094599769586889E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0039796213648733685); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044946657497549475); MoveCostFunction : (multiplier=7.0, imbalance=0.11722222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0039796213648733685); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,807 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023700642:82 , srv1798785208:82 , srv106397765:83 , srv1862566385:83 , srv2007013775:83 , srv2104469115:83 , srv816801523:83 , srv1218629041:84 , srv1303617665:84 , srv1486841855:84 , srv1582573339:84 , srv1640059902:84 , srv1692048201:84 , srv2002406083:84 , srv2123255684:84 , srv448188414:84 , srv826798251:84 , srv862008242:84 , srv926088466:84 , srv1077731035:85 , srv109880554:85 , srv1308984770:85 , srv1322442414:85 , srv1327982819:85 , srv1367221697:85 , srv156604371:85 , srv165822613:85 , srv1678237254:85 , srv1846629290:85 , srv2032237776:85 , srv257492856:85 , srv262179332:85 , srv285702461:85 , srv304352780:85 , srv318300952:85 , srv421532963:85 , srv508406925:85 , srv595261913:85 , srv662282987:85 , srv688836406:85 , srv871306257:85 , srv934275245:85 , srv94390347:85 , srv954237402:85 , srv976513261:85 , srv102894050:86 , srv1119789171:86 , srv1300824360:86 , srv1340284044:86 , srv1352546939:86 , srv1371749680:86 , srv1447066897:86 , srv1501159122:86 , srv1517469870:86 , srv1526000588:86 , srv1604970999:86 , srv1637775857:86 , srv1660908087:86 , srv1743271576:86 , srv1768027682:86 , srv1824744940:86 , srv1827416250:86 , srv2007326313:86 , srv2053273618:86 , srv2070006166:86 , srv2089357100:86 , srv2138504829:86 , srv2145338961:86 , srv550588547:86 , srv604843598:86 , srv712878557:86 , srv853699466:86 , srv883937317:86 , srv924324694:86 , srv1245570927:87 , srv1741894218:87 , srv1800139190:87 , srv1847757881:87 , srv187937683:87 , srv1989506446:87 , srv2003894452:87 , srv2025878371:87 , srv2047248170:87 , srv272123373:87 , srv351466036:87 , srv360458112:87 , srv50244372:87 , srv593879846:87 , srv674338024:87 , srv685129980:87 , srv783640872:87 , srv848439245:87 , srv907683209:87 , srv940415693:87 , srv977231113:87 , srv1926466757:88 , srv239003820:88 , srv556731248:88 , srv208248494:89 , srv1027044161:529 }
2024-03-11 05:54:22,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:22,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv106397765=3, srv109880554=5, srv1637775857=26, srv1245570927=8, srv1077731035=4, srv2007013775=47, srv595261913=76, srv1300824360=9, srv2123255684=57, srv783640872=83, srv871306257=89, srv1989506446=44, srv318300952=66, srv1678237254=30, srv1119789171=6, srv50244372=71, srv448188414=70, srv1303617665=10, srv421532963=69, srv2138504829=58, srv2145338961=59, srv272123373=63, srv883937317=90, srv940415693=95, srv1517469870=21, srv862008242=88, srv257492856=61, srv1027044161=1, srv662282987=78, srv593879846=75, srv685129980=80, srv826798251=85, srv351466036=67, srv1322442414=12, srv1743271576=33, srv1926466757=43, srv604843598=77, srv508406925=72, srv1827416250=38, srv304352780=65, srv1582573339=24, srv1604970999=25, srv2053273618=52, srv976513261=98, srv1847757881=40, srv1340284044=14, srv2089357100=55, srv187937683=42, srv285702461=64, srv926088466=93, srv1486841855=19, srv1768027682=34, srv1741894218=32, srv1308984770=11, srv1447066897=18, srv1023700642=0, srv2047248170=51, srv2032237776=50, srv2070006166=53, srv1218629041=7, srv816801523=84, srv1501159122=20, srv2002406083=45, srv1327982819=13, srv2007326313=48, srv2003894452=46, srv102894050=2, srv2104469115=56, srv1660908087=29, srv712878557=82, srv934275245=94, srv1526000588=22, srv1798785208=35, srv1367221697=16, srv1352546939=15, srv1371749680=17, srv954237402=97, srv924324694=92, srv1846629290=39, srv360458112=68, srv239003820=60, srv208248494=54, srv848439245=86, srv1640059902=27, srv1824744940=37, srv853699466=87, srv977231113=99, srv550588547=73, srv165822613=28, srv674338024=79, srv1862566385=41, srv556731248=74, srv688836406=81, srv94390347=96, srv262179332=62, srv156604371=23, srv2025878371=49, srv1692048201=31, srv907683209=91, srv1800139190=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:22,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:22,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-11 05:54:22,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-11 05:54:22,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-11 05:54:22,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-11 05:54:22,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-11 05:54:22,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-11 05:54:22,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-11 05:54:22,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-11 05:54:22,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-11 05:54:22,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-11 05:54:22,906 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0039796213648733685); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044946657497549475); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0039796213648733685); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:22,906 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-11 05:54:22,906 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.020727090674462E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0039796213648733685); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044946657497549475); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0039796213648733685); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-11 05:54:23,132 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 299 ms to try 5000 different iterations.  Found a solution that moves 197 regions; Going from a computed imbalance of 6.020727090674462E-4 to a new imbalance of 1.1049129206431367E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.02188888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:23,132 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1245570927:83 , srv1300824360:83 , srv2070006166:83 , srv550588547:83 , srv688836406:83 , srv853699466:83 , srv883937317:83 , srv1023700642:84 , srv106397765:84 , srv1077731035:84 , srv109880554:84 , srv1218629041:84 , srv1308984770:84 , srv1327982819:84 , srv1501159122:84 , srv1517469870:84 , srv1582573339:84 , srv1637775857:84 , srv165822613:84 , srv1660908087:84 , srv1743271576:84 , srv1768027682:84 , srv1824744940:84 , srv1827416250:84 , srv1862566385:84 , srv1989506446:84 , srv2003894452:84 , srv2007326313:84 , srv2025878371:84 , srv2032237776:84 , srv2047248170:84 , srv208248494:84 , srv2089357100:84 , srv2123255684:84 , srv2138504829:84 , srv2145338961:84 , srv239003820:84 , srv262179332:84 , srv272123373:84 , srv285702461:84 , srv318300952:84 , srv351466036:84 , srv360458112:84 , srv421532963:84 , srv448188414:84 , srv50244372:84 , srv556731248:84 , srv593879846:84 , srv595261913:84 , srv674338024:84 , srv685129980:84 , srv712878557:84 , srv783640872:84 , srv826798251:84 , srv848439245:84 , srv871306257:84 , srv940415693:84 , srv94390347:84 , srv954237402:84 , srv976513261:84 , srv977231113:84 , srv102894050:85 , srv1119789171:85 , srv1303617665:85 , srv1322442414:85 , srv1340284044:85 , srv1352546939:85 , srv1367221697:85 , srv1371749680:85 , srv1447066897:85 , srv1486841855:85 , srv1526000588:85 , srv1604970999:85 , srv1640059902:85 , srv1678237254:85 , srv1692048201:85 , srv1741894218:85 , srv1798785208:85 , srv1800139190:85 , srv1846629290:85 , srv1847757881:85 , srv187937683:85 , srv1926466757:85 , srv2002406083:85 , srv2053273618:85 , srv257492856:85 , srv304352780:85 , srv508406925:85 , srv604843598:85 , srv662282987:85 , srv862008242:85 , srv907683209:85 , srv924324694:85 , srv926088466:85 , srv934275245:85 , srv156604371:86 , srv2007013775:86 , srv2104469115:86 , srv816801523:86 , srv1027044161:649 }
2024-03-11 05:54:23,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:23,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv106397765=3, srv109880554=5, srv1637775857=26, srv1245570927=8, srv1077731035=4, srv2007013775=47, srv595261913=76, srv1300824360=9, srv2123255684=57, srv783640872=83, srv871306257=89, srv1989506446=44, srv318300952=66, srv1678237254=30, srv1119789171=6, srv50244372=71, srv448188414=70, srv1303617665=10, srv421532963=69, srv2138504829=58, srv2145338961=59, srv272123373=63, srv883937317=90, srv940415693=95, srv1517469870=21, srv862008242=88, srv257492856=61, srv1027044161=1, srv662282987=78, srv593879846=75, srv685129980=80, srv826798251=85, srv351466036=67, srv1322442414=12, srv1743271576=33, srv1926466757=43, srv604843598=77, srv508406925=72, srv1827416250=38, srv304352780=65, srv1582573339=24, srv1604970999=25, srv2053273618=52, srv976513261=98, srv1847757881=40, srv1340284044=14, srv2089357100=55, srv187937683=42, srv285702461=64, srv926088466=93, srv1486841855=19, srv1768027682=34, srv1741894218=32, srv1308984770=11, srv1447066897=18, srv1023700642=0, srv2047248170=51, srv2032237776=50, srv2070006166=53, srv1218629041=7, srv816801523=84, srv1501159122=20, srv2002406083=45, srv1327982819=13, srv2007326313=48, srv2003894452=46, srv102894050=2, srv2104469115=56, srv1660908087=29, srv712878557=82, srv934275245=94, srv1526000588=22, srv1798785208=35, srv1367221697=16, srv1352546939=15, srv1371749680=17, srv954237402=97, srv924324694=92, srv1846629290=39, srv360458112=68, srv239003820=60, srv208248494=54, srv848439245=86, srv1640059902=27, srv1824744940=37, srv853699466=87, srv977231113=99, srv550588547=73, srv165822613=28, srv674338024=79, srv1862566385=41, srv556731248=74, srv688836406=81, srv94390347=96, srv262179332=62, srv156604371=23, srv2025878371=49, srv1692048201=31, srv907683209=91, srv1800139190=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:23,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:23,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-11 05:54:23,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-11 05:54:23,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-11 05:54:23,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-11 05:54:23,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-11 05:54:23,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-11 05:54:23,230 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.669705723422084E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:23,231 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1245570927:83 , srv1300824360:83 , srv2070006166:83 , srv550588547:83 , srv688836406:83 , srv853699466:83 , srv883937317:83 , srv1023700642:84 , srv106397765:84 , srv1077731035:84 , srv109880554:84 , srv1218629041:84 , srv1308984770:84 , srv1327982819:84 , srv1501159122:84 , srv1517469870:84 , srv1582573339:84 , srv1637775857:84 , srv165822613:84 , srv1660908087:84 , srv1743271576:84 , srv1768027682:84 , srv1824744940:84 , srv1827416250:84 , srv1862566385:84 , srv1989506446:84 , srv2003894452:84 , srv2007326313:84 , srv2025878371:84 , srv2032237776:84 , srv2047248170:84 , srv208248494:84 , srv2089357100:84 , srv2123255684:84 , srv2138504829:84 , srv2145338961:84 , srv239003820:84 , srv262179332:84 , srv272123373:84 , srv285702461:84 , srv318300952:84 , srv351466036:84 , srv360458112:84 , srv421532963:84 , srv448188414:84 , srv50244372:84 , srv556731248:84 , srv593879846:84 , srv595261913:84 , srv674338024:84 , srv685129980:84 , srv712878557:84 , srv783640872:84 , srv826798251:84 , srv848439245:84 , srv871306257:84 , srv940415693:84 , srv94390347:84 , srv954237402:84 , srv976513261:84 , srv977231113:84 , srv102894050:85 , srv1119789171:85 , srv1303617665:85 , srv1322442414:85 , srv1340284044:85 , srv1352546939:85 , srv1367221697:85 , srv1371749680:85 , srv1447066897:85 , srv1486841855:85 , srv1526000588:85 , srv1604970999:85 , srv1640059902:85 , srv1678237254:85 , srv1692048201:85 , srv1741894218:85 , srv1798785208:85 , srv1800139190:85 , srv1846629290:85 , srv1847757881:85 , srv187937683:85 , srv1926466757:85 , srv2002406083:85 , srv2053273618:85 , srv257492856:85 , srv304352780:85 , srv508406925:85 , srv604843598:85 , srv662282987:85 , srv862008242:85 , srv907683209:85 , srv924324694:85 , srv926088466:85 , srv934275245:85 , srv156604371:86 , srv2007013775:86 , srv2104469115:86 , srv816801523:86 , srv1027044161:649 }
2024-03-11 05:54:23,256 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=258 (was 245) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=12440 (was 12914)
2024-03-11 05:54:23,264 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=258, ProcessCount=174, AvailableMemoryMB=12439
2024-03-11 05:54:23,265 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-11 05:54:23,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-11 05:54:23,273 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1044745343:84 , srv1048932042:84 , srv1119010122:84 , srv1275950690:84 , srv1298270251:84 , srv1333053926:84 , srv1368988919:84 , srv1491638070:84 , srv1499215450:84 , srv1531780485:84 , srv1703510965:84 , srv1880360157:84 , srv1887487820:84 , srv1980213736:84 , srv2109894960:84 , srv281427884:84 , srv425505125:84 , srv525778098:84 , srv619476790:84 , srv659661864:84 , srv675038962:84 , srv731319478:84 , srv73382824:84 , srv803327862:84 , srv8790886:84 , srv911444895:84 , srv957851697:84 , srv963249679:84 , srv995000062:84 , srv1151425468:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-11 05:54:23,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:54:23,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1368988919=7, srv963249679=28, srv619476790=19, srv1044745343=0, srv1887487820=13, srv803327862=24, srv525778098=18, srv1531780485=10, srv995000062=29, srv1119010122=2, srv1491638070=8, srv1151425468=3, srv1333053926=6, srv425505125=17, srv1298270251=5, srv2109894960=15, srv957851697=27, srv1499215450=9, srv281427884=16, srv1980213736=14, srv731319478=22, srv659661864=20, srv73382824=23, srv1703510965=11, srv1880360157=12, srv1275950690=4, srv911444895=26, srv675038962=21, srv1048932042=1, srv8790886=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:54:23,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:54:23,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:54:23,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-11 05:54:23,300 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0611753982924554, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:54:23,301 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912385804946741, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0611753982924554, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-11 05:56:23,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9239237 different iterations.  Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912385804946741 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:56:23,295 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1044745343:90 , srv1048932042:90 , srv1119010122:90 , srv1151425468:90 , srv1275950690:90 , srv1298270251:90 , srv1333053926:90 , srv1368988919:90 , srv1491638070:90 , srv1499215450:90 , srv1531780485:90 , srv1703510965:90 , srv1880360157:90 , srv1887487820:90 , srv1980213736:90 , srv2109894960:90 , srv281427884:90 , srv425505125:90 , srv525778098:90 , srv619476790:90 , srv659661864:90 , srv675038962:90 , srv731319478:90 , srv73382824:90 , srv803327862:90 , srv8790886:90 , srv911444895:90 , srv957851697:90 , srv963249679:90 , srv995000062:90 }
2024-03-11 05:56:23,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-11 05:56:23,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1368988919=7, srv963249679=28, srv619476790=19, srv1044745343=0, srv1887487820=13, srv803327862=24, srv525778098=18, srv1531780485=10, srv995000062=29, srv1119010122=2, srv1491638070=8, srv1151425468=3, srv1333053926=6, srv425505125=17, srv1298270251=5, srv2109894960=15, srv957851697=27, srv1499215450=9, srv281427884=16, srv1980213736=14, srv731319478=22, srv659661864=20, srv73382824=23, srv1703510965=11, srv1880360157=12, srv1275950690=4, srv911444895=26, srv675038962=21, srv1048932042=1, srv8790886=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-11 05:56:23,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-11 05:56:23,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-11 05:56:23,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-11 05:56:23,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-11 05:56:23,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-11 05:56:23,311 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-11 05:56:23,311 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1044745343:90 , srv1048932042:90 , srv1119010122:90 , srv1151425468:90 , srv1275950690:90 , srv1298270251:90 , srv1333053926:90 , srv1368988919:90 , srv1491638070:90 , srv1499215450:90 , srv1531780485:90 , srv1703510965:90 , srv1880360157:90 , srv1887487820:90 , srv1980213736:90 , srv2109894960:90 , srv281427884:90 , srv425505125:90 , srv525778098:90 , srv619476790:90 , srv659661864:90 , srv675038962:90 , srv731319478:90 , srv73382824:90 , srv803327862:90 , srv8790886:90 , srv911444895:90 , srv957851697:90 , srv963249679:90 , srv995000062:90 }
2024-03-11 05:56:23,327 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 258), ProcessCount=174 (was 174), AvailableMemoryMB=13112 (was 12439) - AvailableMemoryMB LEAK? -