Rental Housing: Improvements Needed to Better Monitor the Moving to Work Demonstration, Including Effects on Tenants

The MTW demonstration gives 39 participating public housing agencies the flexibility to use funding for HUD-approved purposes other than housing assistance, such as developing affordable housing; change HUD's tenant rent calculation; and impose work requirements and time limits on tenants. In 2015, Congress authorized the expansion of MTW by adding 100 new agencies. GAO was asked to evaluate the MTW demonstration. GAO examined HUD oversight of MTW agencies, including its monitoring of demonstration effects on tenants.

For this report, GAO reviewed HUD and MTW agency policies and documentation; interviewed officials at HUD and seven MTW agencies (selected based on type of policy changes, size, and geographic diversity); and interviewed tenants served by selected agencies. GAO also conducted a statistical analysis comparing data for MTW and non-MTW agencies on public housing occupancy rates, voucher utilization rates, and program expenses.

The Department of Housing and Urban Development‘s (HUD) oversight of the Moving to Work (MTW) demonstration has been limited. Improving oversight—particularly for information collection and analysis—would help HUD assess what MTW agencies have done, including funding use. HUD took steps to improve oversight and reporting, but GAO found limitations in the following areas:

Workforce planning. While HUD has taken steps to address staffing to oversee the current 39 MTW agencies, HUD has not finalized its workforce planning for 100 agencies to be added to the demonstration. According to a 2015 HUD analysis, a large number of additional staff would be needed for the expansion. HUD officials said field office staff might assume greater oversight responsibilities to fill this gap, but a joint (headquarters-field) oversight structure is not final and HUD's workforce analysis has not been updated to reflect this proposed oversight structure.

Data collection. Due to limited data, HUD cannot fully determine the extent to which demonstration flexibilities affected the performance of MTW agencies, especially in relation to outcomes that affect the number of tenants served—occupancy and voucher utilization rates and program expenses. GAO found that MTW agencies had lower yearly median rates for public housing occupancy and Housing Choice Voucher (voucher) unit utilization and higher yearly median program expenses than comparable non-MTW agencies. The differences may be partly the result of demonstration funding flexibilities, such as the ability to use public housing and voucher funding for purposes such as gap financing for affordable housing (a nontraditional activity). But limitations in HUD data (such as not differentiating expenses for nontraditional activities) make it difficult to fully explain differences in outcomes GAO analyzed.

Oversight of reserves. HUD has not implemented a process to monitor MTW reserves or agencies' plans for such reserves, which led to agencies accruing relatively large amounts of unused funds that could be used for vouchers. According to HUD data as of June 30, 2017, the 39 MTW agencies had more voucher reserves than the 2,166 non-MTW agencies that administer the voucher program combined ($808 million compared to $737 million). Without a monitoring process, HUD cannot provide reasonable assurance that MTW agencies have sound plans for expending reserves.

Monitoring the effect of rent reform, work requirements, and time limits on tenants. HUD is limited in its ability to evaluate the effect of MTW policies on tenants. HUD does not have a framework—including clear guidance on reporting requirements and analysis plans—for monitoring the effect of rent-reform, work-requirement, and time-limit policies. HUD guidance instructs agencies to analyze the impact of their rent reform activities, describe how they will reevaluate them, and develop a tenant hardship policy for such policies (but not for time limits or work requirements). But the guidance does not describe what must be included in the analyses or policies, leading to wide variation in how agencies develop them. Also, HUD does not assess the results of agencies' analyses.

GAO makes 11 recommendations to HUD, which include completing workforce planning, developing processes to track use of funds and monitor agencies' reserves, and developing a framework—including clear guidance on reporting requirements and analysis plans—to monitor effects on tenants. HUD generally agreed with eight of the recommendations and disagreed with three, citing the need for flexibility. GAO maintains the recommendations, as discussed further in the report.

The Moving to Work (MTW) demonstration, administered by the  Department of Housing and Urban Development (HUD), was established  in 1996 to provide statutory and regulatory flexibility to participating public  housing agencies. Of approximately 3,900 public housing agencies, 39  were participating in the MTW demonstration as of September 2017. In  fiscal year 2017, MTW agencies received funding of about $4.3 billion,  which represented more than 17 percent of HUD’s public housing and  Housing Choice Voucher (voucher) programs. The Consolidated  Appropriations Act, 2016 authorized HUD to expand the MTW  demonstration to an additional 100 public housing agencies over 7  years. However, researchers and organizations that advocate on behalf  of residents, including legal aid groups, have raised questions about the  expansion because of the lack of information about the demonstration’s  effects on tenants.

The MTW demonstration is intended to give participating agencies (MTW  agencies) the flexibility to design and test innovative strategies (activities)  for providing and administering housing assistance using funding they  receive for their public housing and voucher programs, including making  changes to HUD’s rent calculation and adopting work-requirement and  time-limit policies for tenants. MTW agencies also are able to combine the  funding they are awarded annually from HUD’s public housing and  voucher programs into a single agency-wide funding source.

In April 2012, we examined issues such as HUD’s monitoring of the MTW  demonstration and potential benefits of and concerns about  demonstration expansion. Our April 2012 findings included that HUD had  not identified what performance data would be needed to assess the  results of the demonstration as a whole and had not established  performance indicators for the demonstration. We also found that  expanding the MTW demonstration could allow agencies to develop more  activities tailored to local conditions, but data limitations and monitoring  weaknesses would make it difficult for Congress to know whether an  expanded MTW demonstration would benefit the additional agencies and  the residents they serve.

You asked us to conduct another review of the MTW demonstration, with  a focus on how the demonstration affected tenants. This report examines  (1) HUD oversight of MTW agencies, including agency reporting and  compliance with demonstration requirements; (2) any association  between MTW flexibilities and program outcomes, including public  housing occupancy rates and voucher unit utilization rates; and (3) the  extent to which HUD monitored effects of rent-reform, work-requirement,  and time-limit policies on tenants.

To examine HUD’s oversight of MTW agencies, we reviewed our 2012  report on the MTW demonstration, the standard agreement that governs  the participation in the demonstration of the existing 39 MTW agencies,  and HUD’s guidance on agency reporting and demonstration  requirements. We interviewed HUD officials about the processes HUD  uses to review the agencies’ annual reports and assess compliance. We  reviewed workforce analyses on the MTW demonstration. We interviewed  HUD officials about their resource needs and plans to monitor current  MTW agencies and any new agencies that might join the demonstration.  We compared HUD’s monitoring guidance with federal internal control  standards and key principles we developed for workforce planning. To  assess the extent to which HUD followed its procedures, we reviewed  HUD’s documentation of compliance assessments from 2013 through  2016 (the only years for which HUD completed such assessments). We  also interviewed officials from a nongeneralizable sample of seven MTW  agencies that had implemented major rent-reform changes and work- requirement and time-limit policies. In selecting the agencies, we also  considered size, length of time in the demonstration, and geographic  diversity.

To identify and examine any association between MTW flexibilities and  program outcomes, we obtained the following data on MTW and non- MTW agencies for 2009 through 2015: agency and tenant characteristics  from the Public and Indian Housing Information Center (PIC) system;  public housing occupancy rates from the Picture of Subsidized  Households dataset; voucher unit utilization rates from the Voucher  Management System (VMS); and expense data from the Financial Data  Schedule (FDS). These were the most reliable and recent data available  at the time of our analysis. We combined the HUD data with data from the  American Community Survey (1-year estimates) conducted by the  Census Bureau. To assess the reliability of the data, we reviewed  relevant documentation on the information systems, conducted electronic  testing, and interviewed officials knowledgeable about the data. We  determined the data were sufficiently reliable to identify a comparison  group and compare the outcomes of certain measures for MTW and  comparable non-MTW agencies. We used the data and multivariate  statistical methods to compare MTW and similar non-MTW agencies to  estimate any association between MTW flexibilities and public housing  occupancy rates, voucher unit utilization rates, and various public housing  and voucher expenses. To identify factors that could explain the results of  our analysis, we reviewed the standard agreement, FDS data, and data  on unspent voucher funds, and interviewed HUD officials.

To determine the extent to which HUD monitored the effect on tenants of  rent-reform, work-requirement, and time-limit policies, we reviewed HUD  guidance to determine how HUD defines these activities and what  direction it provides on monitoring and reporting the effects on tenants.  We compared HUD’s monitoring guidance with relevant internal control  standards. We reviewed MTW agencies’ 2015 annual reports to  determine the extent to which agencies adopted rent-reform, work- requirement, and time-limit policies. We selected 2015 because it was the  most recent year for which annual reports were available for all MTW  agencies at the time of our analysis. We also reviewed MTW agencies’  2011–2016 annual plans and requested information from all MTW  agencies on tools they use to monitor the effects of rent reform on  tenants. (We chose this range because the 2011 annual plans were the  first to require that all MTW agencies include specific information when  proposing rent-reform policies and the 2016 plans were the most recent  year available for all MTW agencies at the time of our analysis.) We also  interviewed officials from the seven selected MTW agencies about their  monitoring of rent-reform, work-requirement, and time-limit policies’  effects on tenants. Additionally, we held group meetings with tenants from  five agencies and interviewed tenant advocacy organizations to obtain  their views on the effect of these policies on tenants and their awareness  of associated hardship policies. For more detailed information on our  scope and methodology, including how we selected tenants and tenant  advocacy organizations, see appendix I.

We conducted this performance audit from February 2016 to January  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

The MTW demonstration was authorized by the Omnibus Consolidated  Rescissions and Appropriations Act of 1996 (1996 Act). The  demonstration’s ultimate goal is to identify successful approaches that  can be applied to public housing agencies nationwide. As of November  2017, a total of 39 agencies were authorized to participate in the  demonstration (see fig. 1); however, two agencies consolidated their  MTW demonstration programs and are counted as one agency for  purposes of MTW participation.

The MTW Office within the Office of Public and Indian Housing (PIH) is  responsible for implementing the demonstration. The MTW Office  currently includes a program director and eight coordinators, who are  each assigned to a specific group of MTW agencies. MTW coordinators  facilitate the reviews of planned and implemented activities and are  responsible for coordinating with other HUD offices, including local HUD  field offices, to obtain additional input on MTW agencies’ planned  activities and accomplishments.

Objectives and Key Demonstration Requirements

The 1996 Act that created the MTW demonstration provides three  objectives for the demonstration: (1) reduce costs and achieve greater  cost-effectiveness in federal housing expenditures; (2) give incentives to  families with children where the head of household is working, seeking  work, or is preparing for work by participating in job training, educational  programs, or programs that assist people to obtain employment and  become economically self-sufficient; and (3) increase housing choices for  low-income families.

In making these changes, MTW agencies must comply with the following  five contractual requirements derived from the 1996 Act:  1.  assist substantially the same total number of eligible low-income  families under MTW as would have been served absent the  demonstration;  2.  maintain a mix of families (by family size) comparable to those they  would have served without the demonstration;  3.  ensure that at least 75 percent of the families assisted are very low- 4.  establish a reasonable rent policy to encourage employment and self- 5.  assure that the housing the agencies provide meets HUD’s housing  quality standards.

MTW agencies do not receive special funding allocations; rather, they  receive funds from the three traditional primary funding sources (public  housing capital funds, public housing operating funds, and voucher  funds). Public housing agencies generally are required to use the funds  from each source only for specific purposes, but MTW agencies may  combine the money from the three sources and use the funds for a  variety of HUD-approved activities. This fungibility is intended to give  MTW agencies greater flexibility. For example, public housing operating  funds are traditionally used to make up the difference between rents  charged for units and the cost of operating them. Capital funds are  traditionally used for modernization and management improvements,  while voucher funds traditionally provide rental assistance in the private  market. However, an MTW agency may use public housing capital funds  to issue additional vouchers or use voucher funds to develop more public  housing. MTW agencies also have the authority to use their funds to  implement innovative activities that differ from traditional housing  assistance. For instance, an MTW agency can use funds to replace public  housing with mixed-income communities or reach special-needs  populations using vouchers paired with supportive services.

Terms of Participation for MTW Agencies, Including Reporting

HUD entered into a standard agreement with each existing MTW agency.  HUD created the agreement in 2008 to standardize the contract terms.  The agreement references an attachment that sets out reporting  requirements (Attachment B or Form 50900) and another attachment  (Attachment C) that lists the specific sections of the United States  Housing Act of 1937, as amended, and its implementing regulations that  an MTW agency may waive as part of its MTW flexibility. While the  standard agreement is generally the same for all MTW agencies, two  attachments are tailored to individual agencies: a description of the  formulas for determining the amounts of funding each agency will receive  (Attachment A) and a section that may include some agency-specific  authorizations (Attachment D).

In addition to statutory requirements, the agreement requires all existing  MTW agencies to submit to HUD an annual plan for approval as well as  an annual report. Attachment B outlines the information that agencies  are required to include in their annual plans and annual reports. For  example, MTW agencies must include certain elements in their annual  plans for each activity they propose to adopt, such as (1) a description of  the activity and its anticipated effect in relation to the statutory objective  under which the activity is proposed; (2) the HUD metrics that will be used  to quantify the changes the agency anticipates as a result of the activity, including baseline performance level and yearly benchmarks; and (3) the  MTW authorizations that give the agency the flexibility to conduct the  activity. Similarly, MTW agencies are required to include in their annual  reports information about housing stocks and leasing as well as  information required for HUD to assess compliance with key  demonstration requirements (such as number and mix of families served  and percentage of very low-income households served). For rent-reform  activities, agencies are also required to describe the number and results  of any hardship requests.

MTW agencies also are required to report standard information through  HUD data systems. MTW agencies must submit tenant-related data into  the Moving to Work section of the Public and Indian Housing Information  Center (MTW-PIC). According to HUD officials, the MTW-PIC module  was created in 2007 because the standard PIC system that non-MTW  agencies use could not accommodate some of the activities allowed  under MTW, such as rent calculations that vary from HUD’s standard  calculations. MTW agencies also must submit year-end financial  information into FDS, and HUD issued special instructions to enable  MTW agencies to complete the reporting. Finally, MTW agencies must  report voucher unit utilization in VMS.

The Consolidated Appropriations Act, 2016 authorized HUD to expand  the MTW demonstration from the current 39 public housing agencies to  an additional 100 agencies (expansion agencies) over 7 years. The  2016 act requires that the expansion agencies must be high performers at  the time of application and that the selected agencies represent  geographic diversity. The expansion agencies will be brought into the  demonstration by cohort, as required by the 2016 act. HUD plans to  designate the initial cohort by summer 2018. As directed by the 2016 act,  within each cohort each agency will implement one policy change that  HUD selects for that cohort to test. The 2016 act requires that expansion  agencies be rigorously evaluated and that HUD establish a research  advisory committee to advise the Secretary on policies to study and  methods of research and evaluation. HUD established the committee and  received its recommendations on which policy changes to test and how to  evaluate them. As of November 2017, HUD had not announced the  policy changes each cohort will be testing.

On January 23, 2017, HUD published in the Federal Register a request  for comment on a draft operations notice for the MTW expansion. The  draft operations notice establishes requirements for the implementation  and continued operations of the demonstration and describes waivers  available, terms of participation, funding and financial reporting, and  administration and oversight for agencies joining under the expansion.  The comment period closed on June 5, 2017. According to HUD officials,  there will be another opportunity for comment before the notice is  finalized in early 2018.

HUD Took Steps to Improve Oversight, but Has Not Conducted Workforce Planning for Demonstration Expansion

Since our last review of the MTW demonstration in April 2012, HUD has  taken steps to improve MTW agencies’ annual reporting and its process  for monitoring agencies’ compliance with requirements of the  demonstration. However, we found that HUD’s oversight—review of  annual reports and compliance assessments—has not been timely and  HUD has not fully documented its process for assessing compliance, due  to limited staffing and competing priorities. While the MTW Office added  staff to assist with the oversight of the current 39 MTW agencies, HUD  has not conducted workforce planning to address the resources needed  for overseeing the 100 agencies to be added through the MTW  demonstration expansion.

HUD Took Some Steps to Improve Reporting by MTW Agencies and Its Process for Monitoring Compliance

HUD has taken steps to improve MTW agencies’ annual reporting. While  agencies were already required to submit annual plans and reports, HUD  revised its reporting requirements for MTW agencies in May 2013 in  response to our recommendations. Specifically, HUD revised  Attachment B to provide detailed clarifications on the meaning of the  three statutory objectives of the demonstration and relevant standard  metrics. For example, for each of the statutory objectives, the revised  guidance requires that the MTW agency use and report on all of the  applicable standard metrics listed in Attachment B. The revised  attachment also includes standard tables for MTW agencies to provide  operating information and financial information. Additionally, HUD  conducted training on the revised Attachment B and issued a document  containing answers to frequently asked questions about the revisions.

HUD also took some steps to improve its monitoring of MTW agencies’  compliance with the five requirements of the demonstration. Specifically,  in response to our 2012 recommendation that HUD implement a process  for assessing compliance with the requirements, HUD developed a  process and began to track MTW agencies’ compliance with each of the  five requirements. The 2013 revisions to Attachment B added  requirements for agencies to submit information in annual reports with  which HUD assesses compliance. The attachment includes standard  tables for MTW agencies to provide specific information on households  served, family sizes, and income levels.

According to our review of HUD documents and discussions with HUD  officials, the MTW Office uses this information, along with information  MTW agencies submitted in other HUD data systems, to assess  compliance with the five requirements. Table 1 summarizes HUD’s  description of its compliance processes for each of the five requirements.

HUD’s Monitoring Was Not Timely and Its Process for Assessing MTW Agencies’ Compliance Was Not Well Documented

Annual Report Review and Compliance Assessment Timeliness

We found that HUD’s reviews of MTW agencies’ annual reports were not  completed in a timely manner; reviews were completed multiple years  after the annual reports were submitted. Specifically, HUD did not  complete its review of the agencies’ 2013–2015 reports until March 2017  and its review of 2016 reports was still underway as of November 2017  (see fig. 2).

As previously mentioned, MTW agencies submit information about their  MTW activities, financial information, data related to compliance  assessments, and other information through annual reports. Attachment  B states that HUD officials will use this information to monitor MTW  agencies, particularly their compliance with some of the five  requirements. Although the standard agreement gives MTW agencies 90  days after the end of their fiscal year to submit the annual report to HUD,  it does not specify a time frame for HUD’s review of the report. However,  it states that HUD must notify an agency in writing if it requires additional  information or clarifications to the information provided in the report.

HUD officials said that limited staffing resources in the MTW Office in  2014–2016 led to delays in the reviews. Officials further noted that in  2014 and 2015 existing staff in the MTW Office had to focus on other  priorities, including renegotiating the standard agreement, and then in  2016 on implementing the expansion of the demonstration. Untimely  reviews of MTW annual reports diminishes oversight and can result in  delays on HUD’s part in responding to issues arising from the review,  agencies not having an opportunity to respond to concerns promptly, and  HUD’s inability to assess the information reported to determine effects on  tenants.

As previously described, HUD developed a process to assess compliance  with the five requirements of the demonstration, but its implementation of  the process was not always timely. HUD did not complete its 2013–2015  reviews of MTW agencies’ compliance with the five requirements until  2017. In March 2016, HUD officials provided us with a tracker of  agencies’ compliance with the requirements that indicated HUD started its  review for 2013 but had not yet completed that assessment or started  reviewing compliance for subsequent years. In July 2017, HUD provided  us with evidence it had completed the 2013–2016 assessments for all five  requirements.

Documentation of Compliance Assessment Process

In addition, HUD has not clearly documented its process for assessing  compliance with the five requirements. HUD officials told us they did not  have documentation of the process they used to assess compliance with  most of the requirements, such as the methodologies and data used. As  previously discussed, HUD has different processes for assessing  compliance with each requirement and the information it uses to  determine compliance comes from various data sources. Based on our  review of HUD documents (including Attachment B and the recently  completed compliance assessments) and discussions with HUD officials,  it was not always clear what methods HUD used to support its  compliance determinations. For example, documentation we reviewed on  the requirement that MTW agencies ensure that 75 percent of the  households served are very low-income did not state the methodology  used to determine if MTW agencies were in compliance. While our review  of the documentation indicated that tenant income in all relevant  programs was used, it was not clear if the percentages of tenants in each  income category were averaged or weighted to obtain the final  percentage of tenants with very low incomes.

Additionally, while Attachment B briefly describes the data sources used  for some of the compliance assessments, HUD has no documentation  specifying what data variables to extract and how to use them. The lack  of written instructions led to HUD having to redo its assessment of  compliance with the requirement that MTW agencies ensure that 75  percent of the households served are very low-income. Specifically, HUD  officials noted that HUD staff initially determined compliance with this  requirement based on tenants’ current income, but later determined that  they needed to reassess compliance with the requirement using tenants’  income at the time of entry to the program. In September 2017, HUD  officials said they were developing internal standard operating procedures  to document their approach to assessing compliance with each  requirement, and expected to complete the procedures by early calendar  year 2018. However, because HUD has not finalized these standard  operating procedures, it is unclear whether they fully document the steps  and data needed to complete the compliance assessments.

Federal internal control standards state that management should develop  and maintain documentation of its internal control system, including for  controls related to any compliance objectives of the agency. They note  that effective documentation assists in management’s design of internal  control by establishing and communicating purposes, roles and  responsibilities, and specifics of implementation to agency staff.

HUD officials stated that limited staffing in the MTW Office in 2014–2016  and competing priorities led to delays in compliance assessments and  development of full documentation on procedures. Limited documentation  for assessing compliance can lead to inconsistent monitoring of MTW  agencies’ compliance with the five requirements. For example, as  previously discussed, the lack of documentation on the process and data  needed led to the need to reassess compliance with the requirement that  MTW agencies ensure that 75 percent of the households served are very  low-income.

HUD Has Not Yet Completed Workforce Planning for the MTW Demonstration

While HUD has taken some steps to address oversight and staff  responsibilities for an expanded demonstration, it has not conducted  workforce planning for the expanded demonstration. Federal internal  control standards state that management should design control activities,  including management of human capital, to achieve objectives and  respond to risks. Management is to continually assess the knowledge,  skills, and ability needs of the entity so that the entity is able to obtain a  workforce that has the required knowledge, skills, and abilities to achieve  organizational goals. In previous work on human capital, we identified key  principles for effective strategic workforce planning, including determining  the critical skills and competencies needed to achieve current and future  programmatic results and developing strategies that are tailored to  address gaps in number, deployment, and alignment of human capital  approaches for enabling and sustaining the contributions of all critical  skills and competencies.

In 2014, the MTW Office engaged in a workforce analysis exercise to  determine staffing levels needed to oversee the MTW demonstration as  configured at that time. Based on the 2014 analysis, the MTW Office  determined that seven staff were needed to oversee the 39 participating  agencies. In 2014, the MTW Office had four staff and in 2015, five (see  table 2). Officials told us that in 2016, an additional five staff were hired in  the MTW Office and that one staff member would focus on financial  analysis and compliance assessment. In 2017, the MTW staff count was  nine. In July 2017, officials told us that based on the 2014 workforce  analysis, they determined they had sufficient resources to oversee the  current 39 MTW agencies.

In response to a congressional request to determine resource needs for  MTW expansion, in December 2015 the MTW Office updated its 2014  workforce analysis. As with the 2014 analysis, the 2015 workforce  analysis discussed the level of staffing resources needed and not the skill  sets and competencies needed to oversee the expanded MTW  demonstration and actions to fill any gaps. According to this analysis,  HUD determined that a significant number of staff would be needed to  oversee the new agencies. Specifically, 41 full-time equivalent personnel  across various HUD offices would be needed to meet the resource needs  of the expansion in 2016–2020.

In September 2017, HUD officials said that because of the current budget  environment, the agency planned to address the staffing gap identified in  the 2015 analysis by developing a joint oversight structure between the  MTW Office and PIH’s Office of Field Operations. According to HUD  officials, currently the MTW Office is primarily responsible for monitoring  MTW agencies (reviewing annual plans and reports and assessing  compliance with demonstration requirements). Field office staff in PIH  assist with the review of MTW agencies’ overall financial health and  public housing occupancy and voucher leasing information, among other  things. HUD plans to continue to follow this oversight structure for the  existing 39 agencies, but have field office staff assume more  responsibilities for agencies that will join the MTW demonstration as a  result of the expansion. MTW Office officials said they have been having  internal discussions through a working group with field office staff in PIH  to discuss the new oversight structure and determine how best to meet  resource needs associated with the expansion. However, as of November  2017, the MTW Office and PIH had not completed plans for joint oversight  of the expanded MTW demonstration with the field offices or assessed  the knowledge, skills, or abilities needed to implement this new oversight  structure. As previously stated, the first cohort of public housing agencies  will join the expanded MTW demonstration by summer 2018.

MTW Office officials also told us that PIH is planning to finalize a  workforce plan by early calendar year 2018 that will address the broad  resource needs of PIH. However, according to MTW Office officials, PIH  has not yet determined the extent to which the human capital resource  needs for the MTW expansion will be incorporated into the PIH workforce  plan. Without strategic workforce planning that reflects the oversight  strategy for the expanded MTW demonstration, identifies the critical skills  and competencies needed, and includes strategies to address any gaps,  HUD will not be able to reasonably ensure that it has the staffing  resources necessary to oversee an expanded demonstration.

Data Limitations Hinder Analysis of MTW Flexibilities, and Outcomes and MTW Reserve Levels Raise Questions

We found significant differences between MTW agencies and comparable  non-MTW agencies in key outcomes: MTW agencies had lower public  housing occupancy rates, lower voucher unit utilization rates, and higher  program expenses in 2009–2015 than similar non-MTW agencies. MTW  funding flexibilities may partly explain the differences, but limitations in  HUD data (such as the inability to determine which funding source was  used to fund which activity) make it difficult to more fully understand the  differences. MTW agencies accumulated relatively large reserves of  voucher funding, but HUD has performed limited oversight of reserves for  these agencies.

MTW Agencies Had Lower Public Housing Occupancy and Voucher Utilization Rates and Higher Expenses Than Comparable Non-MTW Agencies in Recent Years

We found significant differences between MTW agencies and comparable  non-MTW agencies in key outcomes of the public housing and voucher  programs, possibly affecting the number of tenants MTW agencies  served. MTW agencies had lower yearly median public housing  occupancy rates in fiscal years 2009–2015 than comparable non-MTW  agencies, and the difference was statistically significant (see fig. 3). The  median share of public housing units occupied (public housing occupancy  rate) for MTW agencies was 3 percentage points lower than for similar  non-MTW agencies (93 versus 96 percentage points). The middle 50  percent of MTW agencies in our analysis had occupancy rates that  ranged from 88 to 96 percentage points, while the non-MTW agencies in  our analysis had occupancy rates that ranged from 92 to 98 percentage  points.

MTW agencies also had lower rates of voucher unit utilization than  comparable non-MTW agencies in each year during 2009–2015 (see fig.  4). The voucher unit utilization rate for MTW agencies was about 3  percentage points lower than for similar non-MTW agencies (about 93  percent versus about 96 percent). The middle 50 percent of the MTW  agencies had utilization rates that ranged from about 82 to 97 percentage  points, while the non-MTW agencies had occupancy rates that ranged  from about 92 to 98 percentage points.

We also analyzed expenses for the public housing and voucher programs  of MTW agencies and comparable non-MTW agencies in 2009–2015.

For the public housing program, we included all operating expenses the  MTW and non-MTW agencies incurred that were associated with their  public housing properties. As figure 5 shows, median public housing  operating expenses for MTW agencies in each year during 2009–2015  were $7,853 per household and $6,622 for non-MTW agencies, a  difference of about 19 percent. The middle 50 percent of the MTW  agencies had total public housing expenses that ranged from $6,048 to  $11,436, while the non-MTW agencies had expenses that ranged from  $5,827 to $8,355.

We also compared the operating expenses associated with the central  office cost center of MTW and comparable non-MTW agencies. If larger  public housing agencies implement HUD’s property management rules,  they generally are required to create a central office cost center, which  manages all the centralized activities of the agency and earns fees for  providing day-to-day oversight of individual public housing properties  such as property management. As figure 6 shows, median public  housing operating expenses related to the central office cost center for  MTW agencies were about 9 percent higher than comparable non-MTW  agencies in each year during 2009–2015 ($2,745 per household and  $2,520, respectively). The middle 50 percent of the MTW agencies had  central office cost center expenses associated with their public housing  program that ranged from $1,509 to $5,798, while the non-MTW agencies  had expenses that ranged from $1,635 to $4,939 per household.

For the voucher program, we separately examined expenses in 2009– 2015 related to administration, subsidy (housing assistance payments),  and tenant services. MTW agencies had higher median administrative,  subsidy, and tenant services expenses than comparable non-MTW  agencies. As figure 7 shows, median yearly administrative expenses for  MTW agencies were $922 per household and $642 for comparable non- MTW agencies, a difference of about 43 percent. The middle 50 percent  of the MTW agencies had voucher administrative expenses that ranged  from $713 to $1,179, while the non-MTW agencies had expenses that  ranged from $555 to $762.

As shown in figure 8, the yearly median voucher subsidy expenses for  MTW agencies were about 25 percent higher than for comparable non- MTW agencies ($8,295 per household for MTW agencies and $6,629 per  household for non-MTW agencies). The middle 50 percent of the MTW  agencies had voucher subsidy expenses that ranged from $6,128 to  $12,201, while the non-MTW agencies had expenses that ranged from  $5,524 to $8,178.

As shown in figure 9, the tenant services expenses for the voucher  program were higher for MTW agencies than for comparable non-MTW  agencies, and many non-MTW agencies did not record any expenses for  tenant services in HUD’s database for the years we reviewed. These  results are consistent with MTW agencies having more flexibility to use  funds to provide tenant services. The median yearly expenses for tenant  services for MTW agencies were about $37 per household. Although  tenant services are an allowable administrative expense under the  traditional voucher program, more than half of the non-MTW agencies in  our sample did not report any expenses for tenant services for most of the  years we examined. Non-MTW agencies generally use their voucher  funds to make subsidy payments to landlords and for administrative  expenses.

The statistical matching and modeling analysis we conducted improved  upon unadjusted comparisons of MTW and non-MTW agencies, but it  was not designed to estimate the causal effects of MTW flexibilities. To  reduce the influence of known differences between the two groups, we  accounted for broad characteristics that differed between MTW agencies  and non-MTW agencies. However, our analysis did not attempt to  measure the unique circumstances of each MTW agency, but rather  broad outcomes relevant to public housing and voucher programs in  general. For additional details on our methods and results, see appendix  II.

As noted by others who studied the MTW demonstration and our previous  report, no central source of systematic data exists for MTW activities and  outcomes. However, a July 2017 report by Abt Associates, a research  and consulting firm, identified and tested indicators they developed to  track the performance of MTW demonstrations and compare them to  similar non-MTW agencies. As with our analysis, the Abt study found  MTW agencies tended to have worse outcomes than similar non-MTW  agencies on the indicators of voucher administrative expenses and  voucher unit utilization. The study also analyzed other indicators such as  increases in earnings of nonelderly, nondisabled households; households  served by a service coordinator; and share of voucher households in  neighborhoods with lower poverty rates. On many of the other indicators  analyzed, the study found that MTW agencies did better than similar non- MTW agencies. For example, for the self-sufficiency measures examined  in the study, estimates showed that household earnings were more likely  to increase at MTW agencies than at comparison non-MTW agencies.  The study also concluded that MTW agencies were able to serve a  significant number of individuals not reached by traditional housing  assistance and that in many cases, they were also able to offer additional  supportive services. However, because our analysis did not look at these  other indicators, we could not confirm these results.

Limitations in HUD Data Make It Difficult to Fully Understand Differences

Potential Reasons for Observed Differences

The observed differences in public housing occupancy and voucher unit  utilization rates and program expenses between MTW and non-MTW  agencies, which could affect the number of tenants served, may be a  result of MTW agencies’ ability to (1) combine their public housing and  voucher funds and use them interchangeably and (2) use funds to  implement policies that go beyond traditional forms of housing  assistance.

Combined funding and fungibility. The single fund authorization  permits MTW agencies to combine their public housing operating, public  housing capital, and voucher funds into a single agency-wide funding  source and use the funds interchangeably. For instance, voucher funds  may be used for public housing expenses and vice versa, which could  affect utilization and occupancy rates. Our analysis of 2015 data from  FDS, which HUD uses to account for the agencies’ MTW financial data,  showed that 19 MTW agencies transferred voucher funding to their public  housing program as the result of the single-fund authorization (that is,  they transferred more funding to their public housing accounts than they  received through their public housing funding allocation). This analysis  was possible because HUD requires agencies to report financial information in FDS at the public housing project level. However, the  data could not be used to determine whether all the funds transferred to  the public housing accounts were spent on public housing expenses  because, according to HUD officials, FDS is not a system that tracks the  actual drawdown or disbursement of funds. Instead, public housing  agencies use the system to report year-end financial activity. (As  discussed later in this report, FDS data could not be used to determine  the extent to which public housing funds were used for voucher  expenses.)

Nontraditional activities. Public housing occupancy and voucher unit  utilization rates might be lower for MTW agencies in part because MTW  agencies can use funds to implement policies that go beyond traditional  forms of housing assistance. Since October 2009, the demonstration’s  “broader uses of funds” authorization under the standard agreement has  permitted all MTW agencies to adopt local, nontraditional activities, which  HUD guidance organizes into four categories (see table 3).

In July 2017, HUD provided us with data it had recently compiled on the  number of households served through local, nontraditional activities, by  MTW agency, during 2009–2016 (see fig. 10). According to these data,  in 2009 four agencies implemented at least one type of local,  nontraditional housing assistance activity and served 1,177 households (that is, less than 1 household served through local, nontraditional  housing assistance for every 100 MTW public housing and voucher units  available). In 2016, the number of agencies that implemented at least one  local, nontraditional housing assistance activity grew to 25 agencies,  which served 9,787 households (about 2 households served through  local, nontraditional housing assistance for every 100 MTW public  housing and voucher unit available). Some of these households could be  served through a rental assistance program that offers a lower level of  subsidy than is available to households served through traditional  voucher and public housing programs. For example, a local,  nontraditional activity could result in an MTW agency lowering its share of  housing assistance, thereby increasing the tenant’s share of rent.  Conversely, HUD officials pointed out that because MTW agencies assist  hard-to-serve households, the subsidies provided to these households  could be higher than the subsidy provided under HUD’s traditional  housing assistance programs. As such, a household served through local,  nontraditional housing activity may not be equivalent to a household  served under the traditional voucher or public housing program.

Other factors related to expenses. According to HUD officials, factors  that could explain the observed differences in the expenses for the public  housing and voucher programs of MTW agencies and non-MTW  agencies include that MTW agencies typically (1) need more time and  resources to develop and implement “innovative” activities, (2) serve  hard-to-serve households such as those experiencing homelessness, and  (3) provide additional services to the households they serve as a result of  the funding flexibilities. According to a University of North Carolina at  Chapel Hill study, nearly all MTW agencies have used program flexibility  to provide supportive housing for various hard-to-serve populations,  including the previously homeless, mentally disabled, developmentally  disabled, formerly incarcerated, domestic abuse victims, youth aging out  of foster care, and those with substance abuse issues. Some of these  programs were provided through sponsor-based voucher programs  administered by partner agencies, which required coordination between  the MTW agency and the partnering agencies.

Data Limitations Hinder Fuller Explanations

Limitations in HUD data make it difficult to more fully explain the  differences that may affect the number of households served. For  instance, HUD cannot measure how participation in the demonstration  affected the occupancy and voucher unit utilization rates of MTW  agencies. As previously discussed, HUD uses FDS to account for the  agencies’ MTW funds, but once combined in the system, the funds are  decoupled from the original funding source and it is difficult to determine  how these funds were used. As described earlier, although FDS data  could be used to illustrate how many agencies transferred voucher  funding to their public housing program, these data could not be used to  illustrate how many agencies transferred public housing funding to their  voucher program because, according to HUD officials, FDS does not  identify the source of funding that is available for the voucher program  and local, nontraditional activities.

Similarly, FDS cannot measure expenses that were for local,  nontraditional activities because FDS expenditure categories are not  tailored to the MTW demonstration. HUD officials said the reporting of  expenses associated with local, nontraditional activities varies by MTW  agency, which affects where FDS captures such expenses. HUD has  not made changes to FDS because, according to HUD officials, FDS is an  accounting system that tracks agencies’ year-end financial activity and,  therefore, is not designed to keep track of these data.

Furthermore, historical data do not exist on the households served  through local, nontraditional activities. Although HUD provided us a  spreadsheet it compiled in July 2017 with data on the number of  households served through local, nontraditional housing assistance  activities from 2009 through 2016, HUD had to manually compile the  spreadsheet because its PIC system does not capture data on these  households. HUD officials said the agency was considering capturing  some data on local, nontraditional households in PIC, but making this  change would require HUD and MTW agencies to devote resources to  update their systems.

HUD previously considered making changes to the system. In 2012, HUD  issued a Federal Register notice requesting public comment on changes  to the system to track households provided assistance through local,  nontraditional activities. According to the notice, agencies had not been  reporting these families into the system, which made it difficult to  accurately account for the number of MTW families being served. The  notice further stated that the MTW Office was manually collecting data on  the number of families served each year but the PIC system needed to be  revised to make information collection easier for MTW agencies and  HUD. HUD officials said HUD did not have the information technology  resources needed to make this change in PIC.

Federal internal control standards state that management should use  quality information to achieve the entity’s objectives. Additionally, one of  the statutory objectives of the MTW demonstration is to reduce costs and  achieve greater cost-effectiveness in federal housing expenditures, and a  key demonstration requirement is to assist substantially the same total  number of eligible low-income families under MTW as would have been  served absent the demonstration.

As discussed previously, intermingled funding streams, the purpose and  structure of FDS, and limitations in PIC have combined to limit the data  collected and readily available on the MTW demonstration. According to  HUD officials, it would be difficult for HUD to require existing agencies to  report additional financial data because doing so would require changes  to the standard agreement, which generally cannot occur without mutual  agreement between the agencies and HUD. Yet agencies’ specific  reporting obligations are not set forth in the general standard agreement  but rather in Attachment B, which HUD already expanded without  requiring an amendment to the standard agreement in 2011 and 2013  and proposed to do in 2016. The standard agreement states that  agencies must provide in their annual plan the information required in  Attachment B, and under the standard agreement, HUD retains flexibility  to determine what constitutes satisfactory completion of the annual plan.  Further, the standard agreement, which sets forth general covenants for  the demonstration and not specific data points or reporting definitions,  specifically acknowledges that HUD must have the “flexibility to design  and test various approaches” for housing assistance and that the  agencies agree “to cooperate fully with HUD” in the monitoring and  evaluation of the MTW demonstration. Under the standard agreement,  MTW agencies must provide in their annual report “the information  necessary for HUD to assess the Agency’s activities,” without specific  detail. As with the annual plan, HUD retains flexibility to determine what  data agencies must report. Without more comprehensive data on the  uses of MTW demonstration funds and households served through local, nontraditional activities, HUD cannot assess the performance of MTW  agencies in relation to public housing occupancy and voucher unit  utilization rates and program expenses, which could affect the number of  tenants served.

MTW Agencies Had Relatively Large Reserves of Unspent Voucher Funding, but HUD Performed Limited Oversight

MTW agencies have accumulated relatively large reserves of voucher  funding. The agencies are able to accumulate more reserves because  their voucher funding formula differs from the formula used for the  traditional voucher program. HUD allocates voucher funds to non-MTW  agencies based on leasing rates and subsidy costs from the prior year.  As a result, these agencies have an incentive to expend their voucher  funding to keep their budget utilization rate high. However, the voucher  formula for MTW agencies, which is outlined in an attachment to each  agency’s standard agreement, is generally based on the actual, per-unit  costs in the year prior to the agency joining the MTW demonstration.  Because the voucher allocation is not tied to prior-year subsidy expenses,  MTW agencies do not have the same incentive that non-MTW agencies  have to use all their voucher funds in a given year.

According to 2016 HUD voucher reserve data, the 39 MTW agencies had  almost as much voucher reserves as the 2,166 non-MTW agencies  combined. Specifically, as of December 31, 2016, MTW agencies had a  total of about $1.11 billion in voucher reserves, whereas the 2,166 non- MTW agencies had slightly higher reserves of $1.13 billion. Similar to our  analysis above, we compared the voucher reserves MTW agencies held  to the reserves comparable non-MTW agencies held. As figure 11 shows,  as of December 31, 2016, the median amount of reserves per household  held by MTW agencies was $2,462 compared to $480 for comparable  non-MTW agencies (a difference of $1,982 or about 5 times higher). After  we completed our analysis, HUD provided updated reserve levels as of  June 30, 2017, that showed that MTW agencies’ reserves exceeded non- MTW agencies’ reserves. MTW agencies had a total of about $808 million  in reserves while non-MTW agencies had reserves of about $737 million.

HUD has performed limited oversight of MTW reserves. For example,  before 2016 HUD did not capture data that would help it determine the  amount of voucher reserves held by MTW agencies. In January 2012, as  part of a new cash management requirement for the voucher program,  HUD implemented a process to help transition the accrual of excess  funds held at the agency level to HUD-held reserves. According to HUD  officials, this process was only partially implemented for MTW agencies at  that time because voucher subsidy expenses were comingled with  expenses associated with other allowable MTW activities in VMS. In  2016, HUD added new fields in VMS to distinguish various MTW  nonvoucher subsidy expenses (such as those for capital improvements of  existing public housing units and operation of local, nontraditional  activities) from unspent funding. According to HUD officials, these  enhancements to VMS now allow HUD to keep track of MTW agencies’  reserves. Consequently, in 2016, HUD started cash reconciliations for  MTW agencies, consistent with the cash management procedures for  non-MTW agencies.

HUD also does not have a process to systematically determine if MTW  agencies have public housing reserves. Unlike for the voucher program,  HUD was unable to determine the extent to which MTW agencies had  unspent public housing funding in reserves. According to HUD officials,  FDS tracks overall MTW reserves but HUD cannot distinguish between  public housing and voucher reserves because the MTW funds are  combined into a single account and because HUD does not have a  system similar to VMS that separately tracks public housing reserves for  MTW agencies.

According to federal internal control standards, management should  internally communicate the necessary quality information, such as  through written communication, to help achieve the agency’s objectives.  Management should design control activities—policies, procedures,  techniques, and mechanisms—to achieve objectives and respond to  risks. Maintaining comprehensive written policies and procedures will help  ensure that control activities are in place to address risks and carry out  management directives. We also developed criteria—a set of questions— that agency managers and Congress could use to identify and manage  fee revenue instability, including identifying common principles and  leading practices for managing reserve funds. For example, managers  should ask what level of reserves is to be maintained. In addition, they  should consider establishing minimum and maximum reserve levels to  ensure accountability and adherence to the reserve’s goals, justifying the  numbers with program data and risk management considerations. When  established reserve goals have been achieved, such as to fund planned  capital investments, the level of reserve should be assessed for  reasonableness.

However, HUD has not developed and implemented a process to monitor  MTW reserves. Specifically, it does not monitor existing MTW agencies’  reserves to determine what agencies plan to do with these reserves and  assess whether the plans are reasonable given the amount of reserves.  HUD officials said it would require a significant amount of time to  individually compare the MTW agencies’ reserves to their planned  activities. However, HUD officials said that the draft operations notice for  the MTW expansion proposes requiring that expansion agencies hold no  more than 1 year of voucher subsidy funds in reserves. But the notice  did not outline a plan to evaluate whether this cap was appropriate, and  HUD has not yet finalized the notice. Without a process to monitor  existing MTW agencies’ plans for their reserves and the appropriateness  of the cap for expansion agencies, HUD cannot provide reasonable  assurance that MTW agencies have sound plans for expending their  reserves.

HUD Does Not Have a Framework for Monitoring the Effect of Certain Policies on Tenants

HUD does not have a framework—standard definitions for rent reform  and self-sufficiency, clear guidance on reporting requirements, or analysis  plans—for monitoring the effect of rent-reform, work-requirement, and  time-limit policies.

HUD Definition for Rent Reform and Agency- Determined Definitions for Self-Sufficiency Resulted in Inconsistent Reporting and Prevented Data Aggregation

HUD’s definition of rent reform is unclear, leading to agencies  inconsistently categorizing some policies and not reporting required  information for rent-reform policies. Federal internal control standards  state that management should use quality information—relevant and  reliable data—to achieve the entity’s objectives. HUD defines rent  reform as “any change in the regulations on how rent is calculated for a  household.” Under traditional public housing and voucher program rules,  an assisted household generally must contribute the greater of 30 percent  of its monthly adjusted income or the housing-agency established  minimum rent—up to $50—toward its monthly rent. Statute and HUD  regulations direct how public housing agencies are to certify tenant  income and determine a participating household’s tenant rental  payments. Non-MTW agencies must implement this determination  process when a household first joins the program and then on a regular  basis. In addition, the total housing costs, which are used to calculate a household’s tenant rental payment, include both the rent for the unit and  utility costs. As such, an agency is responsible for establishing and  maintaining a utility allowance schedule that provides reasonable  allowances for tenant-paid utilities. MTW agencies can propose rent- reform policies that make changes to these program rules, such as  changing how often tenants are recertified, eliminating certain exclusions  or deductions, or changing the approach agencies use to determine a  household’s tenant contribution.

HUD has 15 categories of activities it considers to be rent reform under  the MTW demonstration, but does not further define the activities under  each category (see table 4). Based on our review of MTW agencies’  2015 annual reports, we identified 194 activities that involved one or more  rent-reform changes based generally on HUD’s categories of rent-reform  activities.

When we requested that agencies provide information on their rent- reform activities, several MTW agencies asked for clarification on how  rent reform was defined and what activities fell into this category.

Based on our analysis of the agencies’ 2015 annual reports, we found  five agencies did not consider 15 of the 194 activities we identified to be  rent reform using HUD’s definition.

Based on our review of the agencies’ 2011–2016 annual plans, we found  that some agencies did not report information they are required to report  when proposing a rent-reform activity in their annual plans.

Based on our review of the 2015 annual reports, we found that 83 of the  194 policies we identified as rent reform did not include any of the  hardship data HUD requires agencies to report for rent-reform activities.

Officials from some MTW agencies said they did not agree with some of  the categories HUD considers to be rent reform. For example, officials  from three agencies told us that they did not consider changes to the  recertification schedule to be rent reform because such changes do not  change how rent is calculated, only the frequency of the calculation.  Officials from one agency said that HUD’s definition did not match their  agency’s definition because the agency restricts its view of rent reform to  any change that affects the actual rent calculation. HUD’s definition  includes any change that affects the process related to rent. Officials from  another agency told us that they believe HUD does not uniformly apply its  definition of rent reform when reviewing agencies’ policies.

HUD officials also told us that they plan to clarify the rent-reform definition  for expansion agencies. But, as noted previously, HUD told us that  making changes for existing MTW agencies could be difficult because  doing so could require changes to the standard agreement, which  generally cannot occur without mutual agreement between the agencies  and HUD. However, HUD’s definition for rent reform is set forth in  Attachment B, which HUD already has revised without changes to the  standard agreement and is currently revising to clarify existing reporting  requirements. Without a more clear definition of rent reform and specific  criteria or standards with which to classify activities as rent reform, HUD  lacks the quality information needed to monitor all rent-reform activities.

Although one of the requirements of the MTW demonstration is to  establish a reasonable rent policy to encourage employment and self- sufficiency, HUD has not defined self-sufficiency, but rather allowed each  agency to develop its own definition. To measure the extent to which  certain MTW activities, including rent-reform activities, encourage  households to achieve self-sufficiency, HUD requires MTW agencies to  report on the number of households that transitioned to self-sufficiency,  among other things. According to Attachment B of the standard  agreement, MTW agencies are allowed to define self-sufficiency for each  activity that is tied to this HUD metric.

MTW agencies’ definitions of self-sufficiency can diverge widely and  sometimes are inconsistent within an MTW agency. Some examples  include defining self-sufficiency as attaining a total gross household income at 80 percent of the area’s paying a minimum rent of $225;  voluntarily terminating housing assistance and other forms of government  assistance; and attaining a household income of 50 percent of the area median income,  even if the family may be receiving other state benefits.

In addition, some agencies use multiple definitions of self-sufficiency. For  example, one agency uses three definitions for self-sufficiency (one for its  public housing minimum rent activity, one for its voucher rent-reform  activity that combined various changes, and another for its public housing  earned income disregard alternative activity).

Previously, we found that clarity, reliability, and balance are three of  several key attributes of successful performance measures, which are  means of objectively assessing the outcomes of programs, products,  projects, or services. A measure has clarity when it is clearly stated and  the name and definition are consistent with the methodology used for  calculating the measure. A measure that is not clearly stated can confuse  users and cause managers or other stakeholders to think performance  was better or worse than it actually was. A measure is reliable when it  produces the same result under similar conditions. Lack of reliability  causes reported performance data to be inconsistent and adds  uncertainty. Another key attribute of successful performance measures is  balance, which exists when measures ensure that an agency’s various  priorities are covered. Performance measurement efforts that  overemphasize one or two priorities at the expense of others may skew  the agency’s performance and keep managers from understanding the  effectiveness of their program.

According to HUD officials, they have not defined self-sufficiency for MTW  agencies because they want to give agencies the ability to address local  needs. However, the individualized definitions have led to measurements  of self-sufficiency that cannot be consistently evaluated across activities  or agencies. In addition, officials said that it would be inappropriate for  them to develop a definition of self-sufficiency for the MTW demonstration  because HUD has not defined it for the department. However, despite the  lack of an agency-wide definition of self-sufficiency, HUD regulations  define self-sufficiency for certain other HUD programs. As such, HUD  also could develop a self-sufficiency definition for the MTW  demonstration. Without a more standardized definition of self-sufficiency  for the MTW demonstration, HUD cannot collect consistent information  that would allow for the evaluation of the effect of MTW rent-reform and  occupancy policies on tenants.

HUD Guidance for Analyses and Reevaluations of Rent- Reform and Hardship Policies Was Not Detailed

HUD’s guidance on how agencies are to perform impact analyses,  reevaluate activities, and establish hardship policies has not described  the elements of the analysis, required submission of reevaluations, or  described elements of hardship policies. Attachment B of agencies’  standard agreement contains general instructions for reporting  information in MTW annual plans and annual reports, including on rent- reform activities. For example, when an agency proposes a rent-reform  activity, the agency must conduct an impact analysis, describe how it will  annually reevaluate the activity, and develop a hardship policy for the  activity. According to HUD officials, HUD implemented these reporting  requirements for rent-reform activities because they could have significant  effects on tenants.

Attachment B suggests agencies take four steps when developing an  impact analysis and include the results, including describing the rent- reform activity and identifying the intended and possible unintended  effects of the activity; however, it does not provide any explanation or  suggestions for how agencies should approach each step. According to  HUD officials, these steps are not required and the only other guidance  provided to agencies to monitor the effect of rent-reform activities is draft  guidance from 2009. The 2009 draft guidance reiterates the four  suggested steps of an impact analysis and provides a narrative  explanation of the purpose of each step along with examples; however,  agencies are not required to follow the guidance and HUD never finalized  it.

We reviewed the impact analyses agencies reported in their annual plans  from 2011 through 2016 and found that agencies’ impact analyses for  their rent-reform policies varied widely in the type of information included  and level of detail. For example, a majority of impact analyses included  whether the activity would increase or decrease tenants’ rent burden and  a majority included other benefits or costs to tenants, but analyses less  often discussed possible unintended consequences of their rent-reform  policies. In addition, some agencies did not include the same type of  information across the analyses of their activities. One agency provided  an example of how a hypothetical tenant’s rent could change when the  agency moved to biennial recertifications, but did not analyze how  tenants’ rent could change for its minimum rent or tiered rent policies.  Another agency included the potential impact on the agency for each of  its proposed activities, but only analyzed the potential rent burden on  tenants for one activity.

In addition, the level of detail included in the impact analyses varied. For  example, in discussing a policy that would change what sources of  income were included in a tenant’s rent calculation, one agency’s impact  analysis stated that the change would save money for tenants. An impact  analysis for a similar policy from another agency included the number of  tenants who would be affected by the policy and a dollar estimate of how  much money tenants could save. Activities that might be considered  administrative, such as changes to the frequency of tenant  recertifications, were less likely to include details such as analysis of the  rent burden on tenants than were other activities. In several agencies’  impact analyses, as well as in interviews with agency officials, agencies  generally indicated that they think of these MTW policies or activities as  being good for tenants, which may explain why agencies were less likely  to discuss burden on tenants.

HUD officials acknowledged the need for more detailed guidance and  said they planned to provide such guidance for the expansion agencies.  HUD officials said that they have not created such guidance for the  existing agencies because they have been focused on the recent  expansion of the demonstration and because doing so could require  changes to the standard agreement. However, the steps for an impact  analysis are contained in Attachment B, to which, under the standard  agreement, agencies must adhere to satisfy their annual reporting  obligations. Further, HUD has already revised Attachment B and  agencies’ reporting requirements contained therein on multiple occasions  without requiring changes to the standard agreement. Officials stated they  could encourage existing agencies to follow the guidance for the  expansion agencies.

Federal internal control standards state that management should  externally communicate the necessary quality information to achieve the  agency’s objectives. By framing the steps in Attachment B as  suggestions and not prescribing the elements of impact analyses, HUD  cannot consistently collect the type of information it needs to assess the  effect of MTW activities on tenants across agencies. For example,  according to HUD officials, one of the purposes of the impact analysis is  to encourage agencies to consider potential unintended consequences of  their activities. However, unintended consequences cannot be assessed  without more detailed impact analyses.

Attachment B does not describe the elements MTW agencies must  include in their annual reevaluation, and HUD does not require MTW  agencies to submit the results of those reevaluations. According to  Attachment B, when agencies propose a rent-reform activity in their  annual plan, they should provide an overview of how they will annually  reevaluate the proposed activity and revise the activity as necessary to  mitigate the negative effects of any unintended consequences. However,  it does not provide any further detail or examples of what agencies should  annually reevaluate. In addition, while HUD requires agencies to perform  annual reevaluations of rent-reform activities, HUD guidance does not  require MTW agencies to report the results of their annual reevaluations.  According to federal internal control standards, management should  externally communicate the necessary quality information to achieve the  agency’s objectives.

Based on our review of agencies’ annual plans submitted from 2011  through 2016, about one-third of the rent-reform policies proposed by  agencies included a description of how agencies planned to annually  reevaluate the policies. The remaining proposals either did not include a  description or agencies stated that they would evaluate the activity  annually without providing further description of how they would perform  the evaluation. When we requested that agencies provide their 2015  annual reevaluations of their rent-reform policies, several of the MTW  agencies were confused about what we meant by annual reevaluation.  Some of those agencies asked if we were referring to their annual report  and one agency asked how an annual reevaluation was different from an  impact analysis.

When we received documentation of what the agencies considered to be  the annual reevaluations of their rent-reform activities, 30 of the agencies  provided us information they are required to include for all of their  activities in their annual reports. For example, agencies must include a  description of their activities and their impact, compare policy outcomes to  HUD metrics, and explain challenges they faced if benchmarks were not  achieved. Most agencies referred us to all or part of this information.  However, some agencies provided analyses that went beyond those  required for annual reports, including evaluations from third-party  researchers. For example, one agency partners with a local university to  conduct an annual survey that allows the agency to assess the effect of  its rent-reform activities on households.

During the course of our work, a HUD official said the agency had not  required MTW agencies to report annual reevaluations because, as long  as agencies had a plan to annually reevaluate their activities and HUD  had the ability to request the reevaluations if concerns arose, HUD did not  want to require agencies to report information HUD did not intend to  analyze. HUD officials later stated that the agency plans to provide more  detailed guidance for the expansion agencies and has been updating  Attachment B to clarify that agencies’ annual reports must include the  results of their annual reevaluations of their rent-reform activities.

In addition, HUD officials said they could issue guidance that encouraged  existing agencies to follow the guidance for the expansion agencies but it  would be difficult to require existing agencies to include specific elements  in these annual reevaluations without changes to the standard  agreement. However, the standard agreement merely requires that MTW  agencies fulfill the annual reporting requirements set forth in Attachment  B, which provides the detailed description of the required elements of the  annual plan and report and which HUD has already revised on multiple  occasions without requiring changes to the standard agreement.

Because HUD allows agencies to determine the process for reevaluating  their activities, most MTW agencies have not collected or reported  additional information on rent-reform activities (including effects or  unintended consequences) outside of the requirements of their annual  reports. This leaves HUD and the agencies themselves less able to  assess the effects of MTW activities on tenants.

While MTW agencies must establish a hardship policy to define the  circumstances under which households may be exempted or receive  temporary waivers from a new rent-reform activity, Attachment B does not  define what elements must be included in the hardship policy. The  nonbinding draft guidance from 2009 we previously discussed suggested  four questions hardship policies should address (including the process  households would use to request an exemption or waiver and how  hardship cases would be resolved).

Officials from the seven agencies we interviewed said they looked to a  range of tools to create their hardship policies. For example, officials from  one agency said they relied on the 2009 draft guidance and officials from  another agency said they relied on Attachment B when developing their  policies. Officials from three other agencies said they reviewed the  hardship policies of other MTW agencies, had conversations with HUD  while planning the activity or waiting for HUD’s review of their annual  plan, or looked to relevant federal regulations. In contrast, officials from  another agency said that there was no guidance available on how to  create their hardship policies because their agency joined the  demonstration the year it began.

Our review of MTW agencies’ hardship policies for rent-reform activities  showed that while these hardship policies had some commonalities, they  also were inconsistent in terms of the type of information included. For  example, of the 84 hardship policies we reviewed, MTW agencies  included a discussion of how the agency processes a hardship complaint  in 56 policies and what remedies are available for residents approved for  a hardship exemption or waiver in 75 policies. In contrast, 26 policies  included information about whether tenants have the ability to reapply for  a hardship exemption or waiver, and 26 policies mentioned if the  agencies have different rules for the elderly or persons with disabilities. In  addition, although most hardship policies generally discussed how a  tenant may claim a hardship and apply for an exemption, some agencies  were much more specific about the process. For example, one agency  stated only that tenants may request a hardship exemption in writing,  while another agency explained which application a tenant needed to fill  out, what supporting documentation to include, and how to submit the  application.

Some agencies have created more parameters around a tenant’s ability  to request a hardship exemption or waiver than others. For example,  some hardship policies are time-limited (that is, tenants have a certain  window of time in which to apply). One agency instituted a hardship policy  for its minimum rent that stated that tenants had 15 days from receipt of  notice of their new household tenant rental payment to apply for a  hardship exemption or waiver. Another agency instituting a hardship  policy for a similar activity did not seem to impose a time limit for a tenant  to request an exemption. In addition, some hardship policies provided  relief for current tenants. For example, one-third of agencies created a  hardship policy for at least one of their activities that either exempted  current residents from the rent-reform activity or provided some form of  temporary relief as the rent-reform policy was implemented.

We also found variation in the information MTW agencies were able to  provide on the households that requested a hardship exemption. We  asked all the MTW agencies to provide us a list of all tenants who  requested a hardship exemption in 2011–2015, including the result of  each request (denied or approved), the current status of each tenant, and  the reason the tenant was no longer receiving housing assistance, if  applicable. Of all the MTW agencies, five said they had not received any  requests for hardship exemptions. Three agencies were only able to  provide us information on those hardship requests that were approved,  two agencies did not indicate if the requests they received were approved  or denied, and one agency did not provide any data because it could not  distinguish hardship requests for its traditional programs from its MTW  activities. Additionally, five agencies did not provide the reasons why  tenants who requested a hardship exemption were no longer receiving  assistance. The remaining 22 agencies were able to provide the  information as requested.

Tenants and advocates expressed mixed opinions about the rent-reform  hardship policies created by the MTW agencies we interviewed.

Some tenants with whom we spoke said they were aware of rent-reform  hardship policies the agencies developed. For example, tenants who  participated in one of our group meetings told us that during their income  recertification the case worker assigned to their case provided them a  checklist that outlined each of the agency’s hardship policies.

When we spoke with advocates who work with tenants subject to MTW  activities, some said most tenants do not know about the hardship  policies available to them. Some tenants and advocates with whom we  spoke said the process for requesting a hardship could be difficult. For  example, one tenant said that although the MTW agency mailed tenants  “frequently asked questions” that described the hardship policy, the  document was confusing and included a citation to the Federal Register  for more information, which was difficult for tenants to access. Advocates  at one organization also said tenants asked for help because the tenants  applied for a hardship waiver through their case manager, but never  received a response. In contrast, during these meetings some other  tenants told us that they had no issues with the hardship policies or the  way in which the MTW agencies implemented them.

As discussed previously, federal internal control standards require  agencies to communicate effectively with external stakeholders to help  achieve agency goals. While HUD’s proposed update to Attachment B  provides more detail than the current version, HUD officials said it could  be difficult to develop more descriptive guidance for existing MTW  agencies because doing so could require changes to the standard  agreement. In addition, officials said they had not been able to develop  more guidance for existing agencies because of their focus on the  expansion demonstration. However, the standard agreement merely  requires that MTW agencies fulfill the requirements contained in  Attachment B, which HUD has already revised on multiple occasions  without requiring changes to the standard agreement. Officials said that  they plan to provide more descriptive guidance for expansion agencies  and encourage existing agencies to follow such guidance. By not  providing more specific direction to the MTW agencies about what to  include in their hardship policies and therefore what is communicated to  tenants, existing agencies may not be adequately communicating all of  the information tenants need to understand the circumstances in which  they may be exempted from rent-reform activities.

HUD Does Not Have Consistent Requirements for MTW Agencies for Rent-Reform, Work- Requirement, and Time- Limit Activities

HUD requirements for MTW agencies that establish policies for work  requirements and time limits are largely inconsistent with requirements  pertaining to rent-reform activities (see table 5). Although HUD has said  it considers work-requirement and time-limit activities to have a great and  direct impact on tenants, the current MTW agencies in the demonstration  are not subject to the same reporting requirements when proposing those  policies as when proposing rent-reform activities. For example, as  previously discussed, HUD guidance in Attachment B requires agencies  to include an impact analysis, annual reevaluation, and hardship policy for  rent-reform activities in their annual plans when the activity is proposed.  However, Attachment B does not include similar requirements for  proposed work-requirement or time-limit policies.

Further inconsistencies include that Attachment C of the standard  agreement, which lists the various MTW flexibilities available to agencies,  requires MTW agencies to create a hardship policy if they establish a  time-limit policy for public housing assistance. However, HUD did not  develop guidance requiring agencies to report on their hardship policies  for time-limit policies for public housing assistance. Furthermore, HUD  does not have a similar requirement for time-limit policies established for  voucher assistance. In addition, in the Federal Register operations notice  for the expansion of the MTW demonstration published in January 2017,  HUD proposed requiring the new MTW agencies to conduct an impact  analysis and develop a hardship policy for rent-reform and time-limit  policies, but develop only a hardship policy for work requirements.

As previously discussed, federal internal control standards require  management to design control activities—policies, procedures,  techniques, and mechanisms—in response to the entity’s risks. In  determining the necessary level of precision for a control activity,  management is to evaluate, among other things, consistency of  performance. A control activity that is performed routinely and  consistently generally is more precise than one performed sporadically.  HUD officials have said that they consider rent-reform, work-requirement,  and time-limit policies to have a great and direct impact on tenants. HUD  was not able to provide an explanation as to why they do not require  similar reporting for all of these activities.

HUD officials said they did not know why MTW agencies were not initially  required to report on impact analyses, annual reevaluations, and hardship  policies associated with work-requirement and time-limit policies in  general. However, they said, currently, these policies are typically  implemented in conjunction with a rent-reform activity so there is still  reporting on the combined policies. HUD officials also stated that if an  agency proposed an activity with a time limit for public housing, the MTW  coordinator reviewing the agency’s annual plan would ensure that a  hardship policy was in place. In addition, when MTW staff review a  proposed work requirement for both the public housing and voucher  programs and a proposed time limit for the voucher program, staff  suggest that MTW agencies adopt hardship policies and conduct impact  analyses for these policies.

HUD officials also stated that the agency plans to require expansion  agencies to develop an impact analysis, annual reevaluation, and  hardship policy for rent-reform, work-requirement, and time-limit policies.  Although HUD officials said it would be difficult to set a similar  requirement for existing MTW agencies because doing so would require  changes to the standard agreement, they stated they could update  Attachment B to incorporate the requirement for a hardship policy for  public housing time limits and develop guidance encouraging existing  agencies to comply with the additional requirements put in place for the  expansion agencies. Without taking these steps, HUD will miss an  opportunity to collect information needed to evaluate the effect of work- requirement and time-limit policies on tenants.

HUD Has Not Incorporated MTW Agency Reporting into Its Monitoring and Does Not Have an Analysis Plan

Although HUD requires MTW agencies to report annually on their rent- reform, work-requirement, and time-limit policies, HUD could not provide  us with documentation of how it analyzed, used, or planned to use the  information it received from agencies on a continuous basis. According to  HUD officials, because of the recently resolved backlog of annual reports,  the MTW Office now can begin to use the years of reported data it  previously had not used.

Officials added they provide the annual plans and reports to other  departments in HUD to conduct ad hoc analysis and that other HUD  offices have used MTW plans and reports when proposing new rules or  legislation related to housing. For example, officials said HUD used MTW  plans and reports when working on HUD’s 2016 rule intended to provide  greater flexibility for agencies administering HUD’s rental assistance  programs. HUD provided us documentation showing that it used  lessons learned from the MTW demonstration to inform legislative  proposals in the agency’s fiscal year 2018 and 2019 budgets. Also, MTW  officials said they intend to use the data in annual reports to inform some  oversight rules.

When asked about the agency’s plan to analyze the information provided  in the annual plans and reports, HUD officials said it had awarded a  contract to the Urban Institute to perform a retrospective evaluation of the  demonstration, and the results will be available in 2018. Officials said  although they have not finalized their reporting requirements for agencies  in the expansion, these agencies likely will not be required to create  annual plans or reports but instead to annually create a supplemental  document to their annual public housing plan. With those agencies,  HUD will be able to learn from each of the cohorts about the effect of a  specific policy being evaluated. However, the plan to analyze the  supplemental documentation and cohorts of the expansion agencies does  not address how HUD plans to use the information it receives from the  current MTW agencies.

Federal internal control standards state that management should  establish monitoring activities and evaluate results. Analysis (evaluation  of results) contributes to the operating effectiveness of monitoring. The  internal control standards also state that management should use quality  information to achieve the entity’s objectives. In doing this, management  is expected to use quality information to make informed decisions and  evaluate the entity’s performance in achieving key objectives and  addressing risks.

Because the MTW Office has not systematically analyzed or evaluated  the information it requires MTW agencies to report—or determined how  best to evaluate it—the agency cannot assess the effect of MTW rent- reform, work-requirement, and time-limit policies on tenants. More  specifically, without a plan for analyzing information in agencies’ impact  analyses, annual reevaluations, and hardship policies, HUD cannot  monitor the effect of rent-reform, work requirement, and time limit policies  on tenants. These limitations also extend to the definitional and guidance  issues we previously discussed. As a result, without a comprehensive  framework—standard definitions, clear guidance on reporting  requirements, and analysis plans—HUD cannot provide assurance that it  is adequately monitoring how MTW activities affect tenants.

The MTW demonstration is on the brink of significant expansion, but HUD  does not yet have the people, data, and processes in place to effectively  oversee agency participants and assess the demonstration’s performance  and effects on tenants.

Workforce planning. Insufficient staffing for the MTW demonstration  already has had negative effects. For instance, HUD has not always  reviewed annual reports that include information needed to determine the  demonstration’s effect on tenants in a timely manner, annually assessed  whether current MTW agencies comply with demonstration requirements,  and fully documented its review processes. When complete, expansion of  the demonstration would more than triple the number of MTW agencies.  By finalizing its workforce planning (including an assessment of  competencies and skills needed) and documenting its compliance review  process, HUD can provide assurance that it would be positioned to  oversee an expanded demonstration before new agencies start being  added in 2018.

Data collection. Our comparison of public housing occupancy and  voucher unit utilization rates and program expenses among MTW and  non-MTW agencies raises questions about agency performance and use  of funding that cannot be fully answered with current data. The  differences among agencies may result in part from the MTW  demonstration’s funding flexibilities. However, HUD is limited in its ability  to readily determine the extent to which MTW funds were used for other  allowable purposes. More comprehensively capturing and tracking data  on uses of funding and the characteristics of households served by local  nontraditional activities would allow HUD to better assess agency  performance. HUD also would be better able to account for differences in  outcomes—especially in relation to occupancy and voucher utilization  rates and program expenses—that affect the number of tenants served.

MTW reserves. The accumulation of relatively large reserves by MTW  agencies also raises questions about funding uses. HUD has performed  limited oversight of MTW voucher reserves and its data and financial  reporting systems are not structured to effectively track public housing  reserves. Developing and implementing a process to monitor MTW  reserves could help HUD provide reasonable assurance that MTW  agencies have sound plans for expending reserves.

Framework for assessing effect of rent-reform, work-requirement,  and time-limit policies on tenants. The effectiveness of certain MTW  activities and their effects on tenants remain largely unknown because  HUD does not have a framework—standard definitions for key terms,  clear guidance on reporting requirements, and analysis plans—for  monitoring rent-reform, work-requirement, and time-limit policies. For  example, the variations in reporting on rent reform and self-sufficiency as  a result of inconsistent definitions of these terms; limited guidance (often  couched as suggestions) HUD provided to agencies for developing  impact analyses, annual reevaluations, and tenant hardship policies; and  inconsistent treatment of rent-reform and work-requirement and time-limit  policies suggest that HUD may have emphasized flexibility to the  detriment of oversight. In addition, HUD does not have a plan for  assessing the information agencies report on the effect of these policies.  Developing such a framework will help both HUD and MTW agencies to  assess performance and determine if activities have advanced  demonstration goals.

We recognize the challenges involved with monitoring the MTW  demonstration, but maintain it is important for HUD to take steps to  achieve and sustain a better balance between flexibility and prudent  oversight. Improving oversight of the demonstration would help HUD  assess what MTW agencies have done, including their use of funding.  Such information also would help inform Congress and the public about  how demonstration innovations have affected tenants.

We are making the following 11 recommendations to HUD:

The Assistant Secretary for PIH should complete workforce planning for  the MTW demonstration to help ensure that PIH has sufficient staff with  appropriate skills and competencies to manage an expanded  demonstration, including reviewing reports and carrying out compliance  reviews in a timely manner. (Recommendation 1)

The Assistant Secretary for PIH should more fully document the process  for annually assessing compliance with the five demonstration  requirements. (Recommendation 2)

The Assistant Secretary for PIH should develop and implement a process  to track how MTW demonstration funds are being used for other  allowable activities, including local, nontraditional activities.  (Recommendation 3)

The Assistant Secretary for PIH should identify and implement changes  to PIC to capture household data for households served through local,  nontraditional activities. (Recommendation 4)

The Assistant Secretary for PIH should develop and implement a process  to monitor MTW agencies’ reserves. (Recommendation 5)

The Assistant Secretary for PIH should clarify HUD’s rent-reform  definition for the MTW demonstration as part of a framework for  monitoring the effect of rent-reform, work-requirement, and time-limit  policies on tenants. (Recommendation 6)

The Assistant Secretary for PIH should set parameters for HUD’s  definition of self-sufficiency for the demonstration, either by providing one  definition or a range of options from which agencies could choose, as  part of a framework for monitoring the effect of rent-reform, work- requirement, and time-limit policies on tenants. (Recommendation 7)

The Assistant Secretary for PIH should revise HUD’s guidance to MTW  agencies to make it clear which elements are required in impact  analyses, annual reevaluations, and hardship policies and the information  required for each element as part of a framework for monitoring the effect  of rent-reform, work-requirement, and time-limit policies on tenants.  (Recommendation 8)

The Assistant Secretary for PIH should develop written guidance for  existing MTW agencies that requires a hardship policy for public housing  time limits and encourages an impact analysis, annual reevaluation, and  hardship policy for work-requirement and time-limit policies for public  housing and voucher programs as part of a framework for monitoring the  effect of these policies on tenants. (Recommendation 9)

The Assistant Secretary for PIH should require an impact analysis,  annual reevaluation, and hardship policy for work-requirement and time- limit policies new MTW agencies adopt for their public housing and  voucher programs as part of a framework for monitoring the effect of  these policies on tenants. (Recommendation 10)

The Assistant Secretary for PIH should develop and implement a plan for  analyzing the information that agencies report on the effect of rent- reform, work-requirement, and time-limit policies on tenants as part of a  framework for monitoring the effect of these policies on tenants.  (Recommendation 11)

We provided a draft of this report to HUD for comment. In written  comments, which are summarized below and reproduced in appendix III,  HUD disagreed with three of our recommendations and generally agreed  with the remaining eight.

In its general comments, HUD made the following points:

HUD noted that our report did not identify any harmful effects on tenants  as a result of MTW flexibilities. As discussed in the draft report, due to  data limitations, we could not evaluate the effect of MTW flexibilities on  tenants. Instead, we focused on the extent to which HUD monitored the  effects of rent-reform, work-requirement, and time-limit policies on  tenants. Furthermore, our analysis of available data showed that MTW  agencies had lower public housing occupancy rates and voucher unit  utilization rates and higher program expenses than comparable non- MTW agencies, which could affect the number of tenants served.

HUD also stated that it seemed we reviewed MTW agencies through the  lens of the traditional housing and voucher programs. HUD noted  fundamental differences in MTW and non-MTW agency operations and  stated it must consider the extensive MTW flexibilities and the locally- designed nature of each MTW agency’s program in administering the  demonstration. HUD stated it did not agree with three of our  recommendations (discussed below) that it noted would restrict an MTW  agency’s ability to exercise MTW flexibility and respond to variations in  local markets. As stated in the draft report, we recognize the challenges  involved with monitoring the MTW demonstration, but maintain it is  important for HUD to take steps to achieve and sustain a better balance  between flexibility and prudent oversight. Furthermore, given that the  demonstration’s ultimate goal is to identify successful approaches that  can be applied to public housing agencies nationwide, we believe we  looked objectively and with the appropriate rigor and contextual  sophistication at MTW agencies.

HUD disagreed with the draft report’s third recommendation to develop  and implement a process to track how public housing and voucher  funding is being used for other allowable activities, including local,  nontraditional activities. HUD stated that funding fungibility and policy  flexibility are the core tenets of the MTW demonstration. As a result,  identifying and tracking expenses paid from a specific funding source are  not necessary and should not be a requirement. We acknowledge the  demonstration’s funding and policy flexibility and did not intend for our  recommendation to be interpreted solely as a suggestion to track funding  sources. We therefore clarified our recommendation to focus on tracking  how MTW demonstration funds are being used for allowable activities,  such as local, nontraditional activities. HUD stated that the revised HUD  Form 50900 or Attachment B (expected to be published in early 2018)  would require existing MTW agencies to estimate the cost of each  planned activity. Although this would provide some cost information, it  would be limited to planned activities only and would not capture actual  costs. Therefore, we continue to believe that more comprehensively  tracking data on uses of funding would allow HUD to better account for  differences in outcomes—especially in relation to occupancy and voucher  utilization rates and program expenses—that affect the number of tenants  served.

HUD disagreed with the fifth recommendation to develop and implement  a process to monitor MTW agencies' reserves. HUD stated that there is  no language in the 1996 Act that limits the reserves of MTW agencies to  a certain level. Although our draft report noted that leading practices for  managing reserve funds include considering establishing a maximum  reserve level, we did not recommend that HUD set such a reserve level  for MTW agencies because we recognized the demonstration’s funding  flexibilities. Rather, we recommended that HUD develop a process to  monitor MTW agencies’ plans for reserves. HUD also commented that by  reviewing and granting approval for all MTW activities that the existing 39  agencies implemented, it already had a process to determine if spending  of reserve funds was reasonable. However, as HUD noted in its  comments on the draft report’s third recommendation, the agency does  not currently require MTW agencies to include the cost of a planned  activity when proposing the activity. An approval process that does not  include a review of information on planned costs, including the extent to  which reserves would be used to fund the activity, is not sufficient  because HUD lacks data needed to determine that reserve expenditures  are reasonable. Finally, HUD noted that PIH’s Financial Management  Division currently tracks the public housing and voucher reserves of MTW  agencies. However, this does not address our concern that HUD does not  monitor existing MTW agencies’ plans for their reserves and whether the  plans are reasonable given the amount of reserves. In order to provide  reasonable assurance that MTW agencies have sound plans for  expanding their reserves, HUD still would have to develop a process to  monitor MTW agencies’ reserves. Therefore, we maintain our  recommendation.

Similarly, HUD disagreed with our seventh recommendation to set  parameters for its definition of self-sufficiency for the demonstration,  either by providing one definition or a range of options from which  agencies could choose. It noted that the MTW demonstration provides  agencies with the ability to develop creative solutions to address local  conditions, and a one-size-fits-all approach is not appropriate. HUD  stated it intentionally has not developed a standard definition for self- sufficiency, because the definition could depend on local conditions such  as employment opportunities and availability of supportive services. We  recognized the need for flexibility in our recommendation by suggesting  that HUD could develop a range of definitions from which MTW agencies  could choose. This approach would provide the necessary flexibility while  still allowing HUD to collect the consistent information needed to evaluate  the effect of MTW rent-reform and occupancy policies on tenants.  Therefore, we maintain our recommendation.

HUD generally agreed with our remaining eight recommendations. For  example, HUD agreed with the draft report’s first recommendation on  workforce planning, but requested that due to the cross-cutting nature of  MTW, we expand the recommendation to include other PIH offices. We  acknowledge that the staff needed to manage the expanded  demonstration may be found outside the MTW Office, and therefore we  modified our recommendation. HUD also agreed with the second  recommendation to more fully document the process for annually  assessing compliance with the five demonstration requirements and said  it will finalize internal written procedures in early 2018. In addition, in  commenting on the fourth recommendation, HUD described plans to  update its data system to capture information on households served  through local, nontraditional MTW activities. Furthermore, in regard to the  eighth recommendation, HUD noted that it plans to develop guidance for  MTW agencies for the monitoring of high-impact activities such as rent  reform, work requirements, and time limits. Finally, in commenting on the  eleventh recommendation, HUD stated it will improve its process of  analyzing the data MTW agencies provide on high-impact activities.

In commenting on our workforce planning finding, HUD made the  following points:

HUD stated that our finding that planning for the MTW expansion  workforce structure has not been completed is not an accurate  characterization. It noted that HUD completed a workforce analysis and  hired five additional staff in 2016 in anticipation of the MTW expansion. In  our draft report, we acknowledged steps that HUD took to increase the  staffing levels of the MTW Office. However, we found that in its workforce  analysis, HUD had not assessed the knowledge, skills, and abilities  needed to implement an oversight structure for the MTW expansion  demonstration. HUD acknowledged in its response to the  recommendation that its workforce planning efforts will continue in 2018.

HUD said our draft report did not discuss two other factors (beyond  insufficient staff) that affected oversight of the MTW demonstration: (1)  2013 was the first year HUD assessed each agency’s compliance with  the five demonstration requirements, and (2) from 2013 to 2015, HUD  was in protracted and complex negotiations with the existing MTW  agencies to determine the terms of the extension of their MTW  participation. Our draft report acknowledged both factors. Specifically, we  noted that HUD developed a process for assessing compliance with the  five demonstration requirements in response to a recommendation in our  2012 report and that the process was implemented in 2013. Our draft  report also stated that HUD officials noted that in 2014 and 2015 existing  staff in the MTW Office had to focus on other priorities, including  renegotiating the standard agreement, and then in 2016 on implementing  the expansion of the demonstration.

HUD said that even with limited staff, MTW agency plans had been  reviewed and approved within the required time frames.

In commenting on our data collection finding, HUD made the following  points:

Related to our multivariate statistical analysis to examine any association  between MTW flexibilities and program outcomes, HUD stated that HUD  and MTW agencies historically found it difficult to establish comparison  groups because MTW and non-MTW agencies implement significantly  different interventions. We agree that comparisons of MTW and non- MTW agencies are difficult to make. We acknowledge that MTW  agencies differ substantially from non-MTW agencies on factors such as  size and market housing costs. Accordingly, we used statistical  techniques to improve on simple comparisons between MTW and non- MTW agencies. These techniques enabled us to identify a group of  comparison non-MTW agencies that were similar to MTW agencies on  important factors such as geographic location, households served, and  county median rents. We then compared outcomes between the two  groups of agencies over a number of years (2009 through 2015). We did  not compare a single MTW agency to a non-MTW comparison group, as  HUD stated. For more detailed information on our analysis, see appendix  II.

HUD also stated that our finding that MTW agencies had higher tenant  services expenses for the voucher program than non-MTW agencies was  an expected outcome (because the demonstration encourages MTW  agencies to engage in employment, self-sufficiency programming, and  tenant services). In our draft report, we stated that the results of the  analysis were consistent with MTW agencies having more flexibility to  use funds to provide tenant services.

Furthermore, HUD said that a comparison of voucher administrative  expenses for MTW and non-MTW agencies was skewed and not a valid  comparison because administrative expenses for MTW agencies  included voucher administrative expenses and other administrative  expenses not permitted under the traditional voucher program.  Differences in financial and performance outcomes that only MTW  flexibilities allow, such as a broader range of administrative expenses,  represent the potential effects of the demonstration, not a source of bias.  The purpose of our analysis was to determine any association between  MTW flexibilities and program outcomes. Because MTW rules allow for  additional administrative expenses, it was appropriate to include these  expenses in our analysis.

In addition, HUD stated that that it had requested the list of the  comparison group of non-MTW agencies to MTW agencies and  suggested the list be included in our report. The agency noted that  without this information, HUD was not able to validate our analysis. As  noted previously, our analysis was not a simple comparison of MTW and  non-MTW agencies. We developed a comparison group, applied  algorithms based on certain assumptions, and conducted sensitivity  analyses that tested these assumptions.  Therefore, simply providing the  list would not enable HUD to reproduce our analysis. Furthermore, we  selected the variables for matching because they were similar across all  agencies in each group (that is, the full distributions), not for any  particular pair of matched agencies. Consequently, we evaluated the  quality of our comparison group using the distributions of these variables  across all agencies in each group. We included those statistics in our  report, rather than the identity of particular agencies, to encourage  systematic evaluations of the matched comparison agencies using  aggregate statistics, rather than anecdotal evaluations of particular  matched pairs. Finally, we communicated with HUD throughout the  review about our data analysis. For example, we met with HUD to  discuss our methodology, provided initial results, and worked with HUD  officials to ensure we were using appropriate data fields.

HUD also provided technical comments, which we incorporated as  appropriate. We considered one comment to be more than technical in  nature. Specifically, in response to our finding that HUD does not require  MTW agencies to submit the results of their annual reevaluations of the  impact of rent-reform activities, HUD officials stated that they consider the  annual report (and information therein) to be the annual reevaluation of  rent-reform activities. However, Attachment B does not include a  requirement that agencies report the results of their annual reevaluations.  Furthermore, if the information currently required to be included in the  annual report satisfied the annual reevaluation requirement, then there  would be no need for HUD to update Attachment B to clarify that  agencies’ annual reports must include the results of their annual  reevaluations, as the agency plans to do. Therefore, we maintain our  finding and made revisions to the report to clarify what is currently  required in Attachment B.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of the Department  of Housing and Urban Development, and other interested parties. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to examine (1) the Department of Housing and Urban  Development’s (HUD) oversight of agencies participating in the Moving to  Work (MTW) demonstration, including agency reporting and compliance  with demonstration requirements; (2) any association between MTW  flexibilities and program outcomes, including public housing occupancy  rates and voucher unit utilization rates; and (3) the extent to which HUD  monitored effects of rent-reform, work-requirement, and time-limit policies  on tenants.

For all our objectives, we interviewed officials from the following seven  MTW agencies:

Boulder Housing Partners (Boulder, Colorado);

Chicago Housing Authority (Chicago, Illinois);

Delaware State Housing Authority (Dover, Delaware);

Lincoln Housing Authority (Lincoln, Nebraska);

Louisville Metropolitan Housing Authority (Louisville, Kentucky);

Housing Authority of the County of San Bernardino (San Bernardino,

San Diego Housing Commission (San Diego, California).

In selecting these agencies, we focused on agencies that had  implemented major rent-reform changes and work-requirement and time- limit policies based on information in a study conducted in January 2015  by the Center for Urban and Regional Studies at the University of North  Carolina at Chapel Hill. We focused on these policies because,  according to HUD, they have a great and direct impact on tenants. We  also considered agency size, length of time in the demonstration, and  geographic diversity. Although the results of the interviews cannot be  generalized to all MTW agencies, they provide insight into the ways in  which agencies implemented MTW flexibilities and report to HUD, among  other things.

In addition, we interviewed representatives of the following research  groups to discuss their recent or ongoing work on the MTW  demonstration: Abt Associates, the Center for Urban and Regional  Studies at the University of North Carolina at Chapel Hill, HAI Group,  Public and Affordable Housing Research Corporation, and the Urban  Institute. We also interviewed representatives of affordable housing  advocacy groups such as the Council of Large Public Housing Agencies;  National Association of Housing and Redevelopment Officials; National  Leased Housing Association; and Public Housing Authorities Directors  Association. Finally, we interviewed resident advocacy organizations such  as the Center on Budget Policy and Priorities, National Housing Law  Project, and National Low-Income Housing Coalition. To select the  groups to interview, we reviewed our 2012 report on MTW, identified  organizations through our background literature review, and obtained  recommendations from those we interviewed.

To examine HUD’s oversight of MTW agencies, we reviewed our 2012  report, relevant HUD policies and procedures, and HUD documentation  relating to compliance with the demonstration. Specifically, we reviewed  the standard agreement that governs the participation of the existing 39  MTW agencies in the demonstration and HUD’s guidance on agency  reporting and the five demonstration requirements. We also interviewed  HUD officials about the processes HUD uses to review the agencies’  annual reports and assess compliance with the demonstration  requirements. We also reviewed workforce analyses and interviewed  HUD officials about their resource needs and plans to monitor the current  MTW agencies and any agencies that may join the MTW demonstration  through its expansion. We compared relevant internal control standards  that apply to federal agencies and best practices we identified for  workforce planning with HUD’s monitoring policies and procedures. To  assess the extent to which HUD follows its processes, we reviewed  HUD’s documentation of compliance assessments from 2013 through  2016, the only years for which HUD had completed such analysis.

To identify and examine any association between MTW flexibilities and  program outcomes, we obtained the following 2009–2015 data on MTW  and non-MTW agencies: agency and tenant characteristics from the  Public and Indian Housing Information Center (PIC) system, public  housing occupancy rates from the Picture of Subsidized Households  database, voucher unit utilization rates from the Voucher Management  System (VMS), and expense data from the Financial Data Schedule  (FDS). These were the most reliable and recent data available at the  time of our analysis. We combined the HUD data with data from the  American Community Survey (1-year estimates) conducted by the  Census Bureau. To assess the reliability of these data, we reviewed  relevant documentation on the information systems, conducted electronic  testing, and interviewed officials knowledgeable about the data. We  determined that the data were sufficiently reliable for the purpose of  identifying a comparison group and comparing the outcomes of certain  measures for MTW and comparable non-MTW agencies.

We used these data and multivariate statistical methods to compare MTW  and non-MTW agencies to estimate any association between MTW  flexibilities and public housing occupancy rates, voucher unit utilization  rates, and various public housing and voucher expenses. We used  statistical matching and modeling methods to identify a comparison group  of non-MTW agencies that closely resembled MTW agencies on  characteristics including number of households served, geographic  location, and housing market characteristics. For more detailed  information on our analysis, see appendix II.

To determine the factors that could partially explain the results of our  analysis, we reviewed Attachment C of the standard agreement to identify  the funding flexibilities the MTW demonstration affords participating  agencies. We also reviewed MTW agencies’ 2011–2016 annual plans to identify the MTW activities that were proposed under those funding  flexibilities and interviewed officials from the seven selected agencies to  learn how they used the funding flexibilities. We started with the 2011  annual plans because that was the first year in which all MTW agencies  were required to include specific information when proposing rent-reform  policies. We ended with 2016 annual plans because it was the most  recent year for which annual plans were available for all MTW agencies at  the time of our analysis.

To illustrate how MTW agencies used their funding flexibility for public  housing, we used FDS data to determine the amount of MTW funds that  were transferred from the Housing Choice Voucher (voucher) program to  the public housing program. To perform this analysis, we compared the  MTW agencies’ 2015 public housing funding—the sum of FDS line items  70600 (HUD public housing agency operating grants) and 70610 (capital  grants)—to the aggregate amount MTW agencies transferred into  individual public housing project accounts. We selected 2015 because it  was the most recent FDS data available at the time of our analysis. We  also reviewed 2009–2016 data from HUD on the number of households  MTW agencies served through their local, nontraditional activities. We  determined that HUD’s process for compiling this information was  sufficiently reliable for our purposes of reporting on local nontraditional activities by tracing 2015 data in the spreadsheet to data in the agencies’  2015 annual reports (the most recent reports available) and interviewing  HUD staff. Finally, we analyzed program data that HUD prepared using  information derived from the Central Accounting and Program System  and VMS on unspent voucher funds as of December 31, 2016, for MTW  agencies and the comparison group of non-MTW agencies.

To determine the extent to which HUD monitors the effect on tenants of  rent-reform, work-requirement, and time-limit policies, we reviewed HUD  documents such as Attachment B of the standard agreement and HUD’s  Table of Applicable Standard Metrics by Activity to determine how HUD  defines these types of activities and the guidance HUD provides on  monitoring and reporting their effects on tenants. As previously  discussed, we compared HUD’s monitoring policies and procedures with  relevant internal control standards. We reviewed MTW agencies’ 2015  annual reports to determine the extent to which agencies adopted rent- reform, work-requirement, and time-limit policies. We selected 2015  because it was the most recent year for which annual reports were  available for all MTW agencies at the time of our analysis. We also  reviewed agencies’ 2011–2016 annual plans and collected information  from all MTW agencies on tools they use to monitor the effects of rent- reform policies on tenants. We reviewed information from all 39 MTW  agencies on their hardship policies and data and their annual  reevaluations of the impact of rent-reform activities. We also collected  information from all MTW agencies on how they monitor the effect of  work-requirement and time-limit policies on tenants. We interviewed  officials from the seven selected agencies about their monitoring of rent- reform, work-requirement, and time-limit policies’ effects on tenants and  associated hardship policies and to obtain their views about HUD  guidance.

We also conducted group interviews with tenants from five agencies to  get their perspective on the effects of rent-reform, work-requirement, and  time-limit policies the agencies had implemented and associated hardship  policies. To select the tenants to invite to these group interviews, we  focused on the populations (for example, those able to work) subject to  these policies. To the extent the MTW agency had a resident advisory  board or comparable resident association, we worked with the boards or  associations to contact tenants. When appropriate, we asked the MTW  agencies to post notices on their websites and throughout their properties  and send mailings to tenants of interest to notify them about the  meetings. Finally, we interviewed representatives from tenant advocacy  organizations. The organizations represented tenants served by four of  the agencies we visited as well as tenants served by two additional MTW  agencies that were not part of the group of seven selected agencies but  that also had implemented major rent-reform changes, work-requirement,  or time-limit policies. We obtained information on the effect of these  policies on tenants and the extent to which tenants were aware of the  hardship policies associated with these policies. To select these groups,  we generally relied on recommendations from a representative of the  National Housing Law Project. For those areas for which a  recommendation was not provided, we identified the local legal aid  association through an Internet search.

We conducted this performance audit from February 2016 to January  2018 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix II: Statistical Analysis of Program Outcomes in MTW and Non-MTW Agencies

We analyzed associations between the Moving to Work (MTW)  demonstration’s flexibilities and two types of outcomes: housing  availability, measured by public housing occupancy and voucher unit  utilization rates, and program expenses, measured by public housing  operating expenses and voucher administrative, subsidy, tenant services  expenses, and voucher reserves per household. These outcomes are  broadly consistent with the goals of the demonstration’s authorizing  statute. MTW was designed to provide flexibility to participating public  housing agencies to design and test innovative strategies, while meeting  certain statutory objectives and demonstration requirements, including  reducing costs and achieving greater cost-effectiveness and assisting  substantially the same number of eligible low-income households as  would have been served absent the demonstration.

In this appendix, we summarize the statistical methods we used to  analyze a dataset we assembled from administrative databases  maintained by the Department of Housing and Urban Development (HUD)  and the American Community Survey (ACS), conducted by the Census  Bureau, to compare MTW and non-MTW agencies on these outcomes.

Our analysis did not seek to conduct a definitive evaluation of the MTW  demonstration’s causal impacts. MTW agencies carry out varied and  unique activities. The agencies also vary widely in size, location, housing  market, and area and tenant demographics—both compared to non-MTW  agencies and among themselves. A persuasive impact evaluation would  need to assess the unique circumstances of each activity and outcome at  each agency.

In contrast, our analysis sought to improve on simple comparisons of  outcomes between MTW and non-MTW agencies, by constructing a  comparison group of non-MTW agencies that were similar to MTW  agencies on variables broadly relevant to housing programs. Although  this multivariate analysis reduced the risk that factors other than MTW  participation may have biased the comparison, we did not seek to hold  constant all factors uniquely relevant to each MTW agency and activity.  As a result, our analysis cannot provide definitive estimates of causal  impacts.

Developing and applying statistical “treatments” to MTW agencies is  complex, due to demonstration rules that allow agencies to conduct  various activities tailored to their unique needs. We considered the option  of forming several groups of MTW agencies, defined by similar activities.  For example, we might have identified all agencies reforming HUD’s rent  calculation formula, and included those agencies in one level of a  multilevel treatment variable. We ultimately rejected this approach due to  limited sample sizes and the difficulty of developing homogeneous groups  of activities. A multilevel approach would have limited the number of  agencies in each level of the treatment. Small sample sizes would have  limited our statistical power to identify differences between treatment  groups, if they existed. In addition, the wide variety of MTW activities  would have made it difficult to produce a sufficient number of  homogenous groups, and would have required subjective judgment about  what activities were sufficiently similar. Instead, we used a binary  treatment measure identifying agencies that participated in MTW or  operated under traditional public housing rules in a given year.

The timing of MTW implementation limited our ability to account for  changes in participation and outcomes over time. Agencies joined the  MTW demonstration at various times between 1996 and 2012, and many  joined before sufficient data became available. Only nine agencies  entered the demonstration after 2009, when HUD’s Public and Indian  Housing Information Center (PIC) system began to provide sufficiently  complete and reliable data on the characteristics of housing agencies we  needed to measure. All agencies that exited the demonstration did so  before 2009. Comparisons within agencies over time can implicitly control  for other factors that may not substantially change before and after  implementation by using data collected before and after agencies joined  the MTW demonstration. We might have been able to implicitly control for  many factors that did not substantially change over short periods, such as  land prices, or that changed in identical ways for MTW and non-MTW  agencies, such as national economic cycles. However, the  implementation of the MTW demonstration and available data limited our  analysis to repeated cross-sectional comparisons of MTW and non-MTW  agencies from 2009 through 2015.

Measuring participation in the MTW demonstration at any one time was  somewhat imprecise. The MTW demonstration was not implemented at  uniform times across agencies, due to variation in the ratification dates of  MTW agreements between HUD and the agency and variation in when  each MTW agency began to implement activities under the  demonstration. For our primary analysis, we classified an agency as  participating in the MTW demonstration if it had ratified an MTW  agreement with HUD at least 1 year before the year measured. In  sensitivity analyses, described below, we assessed how classifying MTW  participants according to different time lags affected our results.

Table 6 lists the number of MTW and non-MTW agencies in our dataset,  based on how MTW participation was defined in the analysis for housing  agencies in the PIC database from 2009 through 2015.

We compared MTW and non-MTW agencies on several outcomes that  are broad measures of housing availability and expenses. The outcomes  were available in HUD data systems and were reliable for our purposes.  However, they do not exhaust the potential outcomes that may be  relevant under the MTW authorizing statute or the design of specific  agency activities. For example, potential outcomes could measure the  number of households that achieve self-sufficiency (as defined by a MTW  agency) or move to a low-poverty neighborhood.

Our specific outcome measures were the following:

Public housing occupancy rate. Occupied units as a percentage of  units available.

Voucher unit utilization rate. Monthly rate of unit months leased divided  by unit months available for the public housing agency.

Public housing operating expenses per household. Total yearly  operating expenses, divided by number of public housing households.

Public housing central office cost center expenses per household.

Total yearly central office cost center operating expenses, divided by  number of public housing households.

Voucher administrative expenses per household. Total yearly  administrative expenses, divided by the number of voucher households.

Voucher subsidy expenses per household. Total yearly expenses for  housing assistance payments, divided by the number of voucher  households.

Voucher tenant services expenses per household. Total yearly  expenses for tenant services, divided by the number of voucher  households.

Reserves per household (2016 only). Unspent voucher housing  assistance funds as of December 31, 2016, divided by the number of  voucher households.

Following the Rubin Causal Model, our primary parameter of interest was  the average (or median) treatment effect on the treated:  where Yij(T) denotes the outcome for agency i at time j in (potentially  counterfactual) treatment condition T. That is, we estimated the expected  difference in outcomes that would exist due to MTW participation, among  those agencies that actually participated in the demonstration.

Estimating the average treatment on the treated is conservative and  appropriate, given the varied and unique nature of MTW activities.  Generalizing the effect of MTW participation from the treated agencies to  the rest of the public housing agency population makes the implausible  assumption that the untreated agencies would have implemented the  same activities, in the same ways, as the treated agencies. Due to the  discretion inherent to the MTW demonstration, the experiences of the  treated agencies may not generalize to the whole population, as would be  required for estimating the average treatment effect.

We specify a parameter of interest (that is, a value to be estimated) for  methodological completeness and to specify the population of inference  (the target population of agencies). However, we do not interpret our  results as robust causal impact estimates, due to the inability to measure  the unique circumstances relevant for each MTW agency, demonstration  activity, and outcome.

Our analysis measured and held constant conditions that could have  otherwise explained differences in outcomes between MTW and non- MTW agencies. For each MTW and non-MTW agency, we measured the  following agency-level covariates (with sources in parentheses):

Percent of households with a member over the age of 65 (PIC)

Percent of households with a member under the age of 18 (PIC)

Percent of households with a disabled member (PIC)

County population density, measured as county population/land  area (2010 Census)

Latitude (Picture of Subsidized Households)

Longitude (Picture of Subsidized Households)

We assessed the reliability of the ACS estimates by calculating the ratio  of each estimate’s 95 percent margin of error to the estimate. For  example, this ratio would equal 5 for an estimated rental vacancy rate of  10 percentage points, with a margin of error equal to plus or minus 2  percentage points. Across all variables we used from ACS, we found that  this ratio did not exceed 2.0 for 99 percent of agency-county  observations. This level of reliability was acceptable for our purposes.

When PIC showed that agencies spanned multiple counties, we  aggregated the data to the agency level by either summing count  variables across counties or calculating averages of ACS descriptive  statistics, such as county mean incomes. We calculated unweighted  averages because the Census Bureau does not release ACS microdata  with the exact geographic locations needed to re-estimate the statistics of  interest within public housing agency boundaries. Weighting by the total  area population or number of households served by each public housing  agency would have had unknown effects on the bias of the published  ACS estimates, due to their complex weighting methods. Our aggregation  methods should minimally influence our measurements, due to limited  variation across counties within agencies. To quantify this variation, we  calculated the coefficient of variation (CV) across counties served by  each agency in our analysis, and these CVs of the ACS statistics did not  exceed 0.99 for 50 percent of the agencies and 1.73 for 95 percent of the  agencies.

We used statistical matching methods to construct the comparison group  of non-MTW agencies. The general iterative matching process involves  1.  identifying some distance measure that quantifies how “close” units  are to each other on the covariates of interest;  2.  implementing a matching method that uses this distance measure to  identify comparison units; and  3.  assessing the quality of the matched samples and iterating between  the first two steps, until the treatment and comparison groups become  sufficiently close on the distance measure.

We developed our specific matching approach using recent reviews of the  statistical literature.

Two established matching methods rely on propensity scores and  Mahalanobis distance (MD). In the context of this analysis, propensity  scores estimate the probability that an agency is an MTW or non-MTW  agency, such as when Pr(MTW | X) = logit-1(Xβ), where X is a matrix of  covariates and β is a vector of coefficients. Propensity scores are  calculated using the estimated coefficients and X to obtain a predicted  probability that an agency participates in the MTW demonstration. MD is  a multivariate sample statistic measuring the distance between agency i  and j, similar to the number of standard deviations away from the sample  mean vector of the covariates:  where X is the ith row vector of X and S is the sample covariance  matrix.

Propensity scores and MD measures can have several limitations in  practice. Matching on known propensity scores is used to balance the  covariate distributions between the treatment and comparison groups and  matching using MD tends to improve balance across all measured  covariates.  However, both approaches are optimal under assumptions  of normally distributed data, and may worsen covariate balance if this  assumption does not hold.

Genetic matching methods seek to solve the problem of achieving sample  balance in practice, using computer algorithms to search over the space  of possible distance measures. Genetic matching generalizes MD by  weighting covariates according to how they achieve balance in any  particular sample, rather than by constants equal to the inverse of their  sample covariance matrix, as in MD:  where W is the covariate weighting matrix. If desired, genetic matching  can incorporate propensity scores by including them as a covariate, with  the algorithm assigning as much weight to them as necessary to optimize  balance.

The genetic matching algorithm, as implemented by the R software  package “Matching,” has the following steps:  1.  Initialize covariate weights, W, at starting values.  2.  Calculate the distance matrix between MTW and non-MTW agencies.  3.  Specify the number of non-MTW agencies to be matched comparison  agencies for each MTW agency.  4.  Assess the balance between the sample distributions of the treatment  and control groups, using p-values from matched t-tests of equal  means for each covariate or Kolmogorov–Smirnov tests of equal  distributions.  5.  Apply a loss function to the vector of p-values to quantify overall  sample balance.  6.  If the loss function is not minimized, regenerate W using a genetic  algorithm.  7.  Repeat steps 2–6 until the loss function is optimized and covariate  balance is maximized.

In sum, the genetic matching algorithm searches for the best k matches,  incorporating covariates and distance metrics as desired and minimizing  the distance in a candidate matched set by weighting and reweighting the  covariates and metrics, according to how they influence balance. In our  primary analysis, we ultimately used one-to-one matching (k = 1), with  one comparison agency selected for each MTW agency. Large  imbalances in the number of households served by the MTW and non- MTW agencies substantially reduced the pool of similar comparison  agencies, such that setting k > 1 substantially worsened the balance for  some variables.

In addition to the automated matching criteria above, we compared the  sample distributions of the covariates before and after matching using  descriptive statistics and nonparametric density estimates. We required  exact matches on the year of measurement to ensure that observations  were compared at roughly the same times. We also required exact  matches on whether an agency issued vouchers and HUD region. Due to  data limitations, we compared 2016 reserve spending between MTW and  non-MTW agencies for the 2015 matched set.

Figure 12 compares MTW agencies and non-MTW agencies on the  covariates we identified, before constructing a matched sample of  comparable non-MTW agencies. As the figure shows, there are some  covariates for which there are significant differences between the group of  MTW agencies and non-MTW agencies.

After implementing the matching method described above, we identified a  primary group of comparison agencies that were similar to the MTW  agencies on most of the covariates, but differed on a few, as shown in  table 7. Examples of matched agencies in our primary analysis include:  Oakland Housing Authority (MTW) and Housing Authority of the County of  Sacramento (non-MTW); San Antonio Housing Authority (MTW) and  Housing Authority of New Orleans (non-MTW); and Housing Authority of  the City of Pittsburgh (MTW) and Allegheny County Housing Authority  (non-MTW). Imbalances between MTW and comparison agencies for the  main analyses remained after our primary matching analysis for county  median income, county median rental cost, number of households,  percent of households with a disabled member, and county rental  vacancy rate, as shown in table 7.

Figure 13 shows the covariate density estimates for MTW and non-MTW  agencies, after matching. As the figure shows, there are fewer differences  in the group of MTW agencies and the matched non-MTW agencies after  matching.

MTW agencies had higher county median incomes and rent, lower  percentages of disabled household members, and lower rental vacancy  rates, as compared to the primary matched non-MTW agencies. These  imbalances decreased when we allowed for matches across HUD region  and required matches within calipers (1 standard deviation), as shown in  table 8. However, allowing HUD region to vary potentially allowed other  unmeasured factors within a HUD region to vary between the MTW and  non-MTW groups. Applying caliper constraints failed to match a  comparison agency for 91 of the 232 yearly observations for MTW  agencies during 2009–2015, which changes the population for inference.  We used these matched samples with improved balance for sensitivity  checks, in our discussion of the results below.

After constructing the primary matched analysis sample, we estimated  outcome descriptive statistics for MTW and non-MTW agencies. We  estimated differences in mean and median outcomes using paired t-tests  and nonparametric Wilcoxon signed-rank tests, respectively, that account  for correlations over time within and between matched groups of MTW  and non-MTW agencies. We estimated differences in medians between  groups using nonparametric Wilcoxon signed-rank tests to address  potential outliers. For example, the tenant services cost distributions for  MTW agencies (median = $37; 25th quantile = $2.80; 75th quantile = $110)  and non-MTW agencies (median = $0; 25th quantile = $0; 75th quantile =  $20) were highly skewed. The nonparametric test was not influenced by  these skewed distributions and outliers.

To complement this matched comparison, we used Generalized Linear  Models to model outcomes in 2009–2015 using the matched sample of  MTW and non-MTW agencies.

The models had the form:  i = 1, …, n indexes agencies  j = 2009, …, 2015 indexes years

MTWij indicates whether agency i participated in the MTW demonstration µij is the mean outcome, conditional on the covariates g is the Gaussian link function

Year is a vector of indicators for each year from 2010 through 2015  (excluding 2009), which accounts for common period effects across  agencies, γ

Xij is a vector of linear continuous (e.g., number of households) and  categorical (e.g., HUD region) control variables that may confound the  association between agency type and the outcome of interest (discussed  above for the matched sample) β is the parameter of interest, estimating the association between MTW  Repeated observations from 2009 through 2015 for MTW agencies and  their corresponding matched non-MTW agencies can introduce  autocorrelation within these clusters of observations, and the differences  across matched clusters can introduce heteroscedasticity (that is, the  variance in one cluster of agencies may be not be consistent with the  variance in another cluster). A conventional linear model does not  account for these interdependencies and inconsistent variances in the  data, leading to potential bias in the variance estimation for the  parameters of interest (such as variances for β and γ) and any  subsequent statistical inference on the association (and p-values)  between the outcome and covariates.

To account for the potential bias arising from heteroscedasticity and  autocorrelation, the variance-covariance matrix used to generate the  variances for the parameters incorporated weights that (1) decreased the  influence of extreme observations, clusters, or both; (2) used an  autoregressive approximation in which the correlation was strongest for  observations closest in time and decays as time lengthens; and (3)  preprocesses (“prewhitens”) the variance-covariance matrix using an  autoregressive function to reduce the temporal dependence in the data.  These processes lead to statistical inference on associations of interest  that account for the interdependencies within agency clusters and the  differences across clusters. In the sensitivity analyses described below,  we will fit this model on the unmatched population of agencies.

In the matched sample, MTW agencies had lower median public housing  occupancy rates and voucher unit utilization rates compared to non-MTW  agencies, as shown in table 9. Compared to non-MTW agencies, MTW  agencies had higher median public housing expenses per household  (operating and central office cost center operating expenses) and higher  median voucher administrative expenses per household, subsidy  expenses per household, tenant services expenses per household, and  reserves per household. These differences were significant at the 0.05  level for all variables using the nonparametric Wilcoxon signed-rank  test. However, using the parametric t-tests and related t-tests from the  regression models, there was not a significant difference in central office  cost center operating expenses. This could arise from the presence of  outliers skewing the distribution, leading to different results compared to  the Wilcoxon test that does not make any distributional assumptions.  Regardless of the particular method used, small sample sizes in each  group, as well as repeated observations over time, may limit our statistical  power to identify differences, if they existed. Sample sizes resulting from  missing data also affect the degree to which comparable non-MTW  agencies can be found, given the limited overlap in the covariate  distributions between groups.

We assessed the results above for sensitivity to various methodological  assumptions. For the matching analysis, we assessed the impact of  1.  measuring MTW status as of the agreement year, rather than as of 1  year following the agreement (i.e., 1 year lag);  2.  matching within 1 standard deviation calipers for each covariate;  3.  allowing matches between HUD regions;  4.  including county unemployment and poverty rates as covariates;  5.  including estimated propensity scores, as a logistic function of the  control variables described for the primary analysis, as a matching  covariate;  6.  increasing the number of comparison agencies for each MTW agency  to k = {2, 3, 4} using the control variables described for the primary  analysis; and  7.  excluding clusters where the MTW and/or non-MTW agencies had an  outlying value for an outcome of interest.

For the regression model, we compared the results obtained from fitting  the model to the matched and unmatched data.

The sensitivity tests above showed no substantively meaningful  differences in the results as compared to the primary analysis, with  several exceptions. Adding the caliper constraint and dropping the HUD  region constraint improved covariate balance. Dropping the HUD region  constraint led to MTW agencies having a smaller difference in voucher  subsidy expenses, compared to non-MTW agencies. In our primary  analysis, MTW agencies had higher subsidy expenses. However,  allowing matches between HUD regions may introduce unmeasured  geographic characteristics into the comparison group of non-MTW  agencies, which may limit the comparability of subsidy expenses and bias  the estimated difference in outcomes.

Appendix III: Comments from the Department of Housing and Urban Development

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Paige Smith (Assistant Director),  Josephine Perez (Analyst in Charge), Enyinnaya David Aja, Bethany  Benitez, Farrah Graham, Anar N. Jessani, Morgan Jones, Courtney  LaFountain, Won Lee, Marc Molino, Anna Maria Ortiz, Barbara  Roesmann, Shannon Smith, and Jeff Tessin made key contributions to  this report.