Meeting — Oct 24, 2023

The GSA Acquisition Policy Federal Advisory Acquisition Workforce Subcommittee convened for a public meeting at 3:00 PM on October 24, 2023, virtually via Zoom, with Nicole Darnall, Chair, presiding.

In accordance with FACA, as amended, 5 U.S.C. App 2, the meeting was open to the public from 3:00 P.M. to 5:00 P.M. EST.

Meeting agenda

Committee Members Present:
Nicole Darnall, Chairperson — Arizona State University
Darryl Daniels — Jacobsen Daniels Assoc.
David Malone — AquireIQ
Kristin Seaver — General Dynamics Information Technology
Clyde Thompson — GovStrive, LLC

Absent: Gail Bassette, Mark Hayden, Anne Rung, Steven Schooner

Guest Speakers & Presenters:

David Gill — Chief, Analytics & Technology Solutions Branch (Acting), Department of the Treasury, Internal Revenue Service, Office of the Chief Procurement Officer

GSA Staff Present:
Boris Arratia — Designated Federal Officer
Stephanie Hardison — Deputy Designated Federal Officer
David Cochennic — GAP FAC Support
Skylar Holloway — GAP FAC Support
Caryn Broome — Closed Captioner
Jill Lamoreaux — ASL Interpreter
Jeffery Bowden — ASL Interpreter

Call to order

Stephanie Hardison, Deputy Designated Federal Officer, opened the public meeting by welcoming the group before reminding the public that there will be time for comments and statements at the end of the meeting. Stephanie then performed a roll call to confirm attendance before turning the meeting over to Chairman Nicole Darnall.

Welcome and opening remarks

Chairman Nicole Darnall welcomed David Gill to a meeting focused on automating decisions within GSA to promote sustainability in federal acquisitions. They’ve been exploring the use of technology to enhance sustainability and have discovered similar efforts in other government agencies, like the Internal Revenue Service (IRS). David Gill, an acting Chief of the Analytics & Technology Solutions Branch at the Internal Revenue Service, is present to discuss his work in implementing environmentally sustainable procurement policies, analyzing contract data related to sustainability, and exploring automation approaches, including artificial intelligence.

Guest speaker and questions

David Gill introduced himself to the subcommittee clarifying that he is no longer signing contracts but has transitioned into an analytics and technology role. He is focused on using innovative technology to advance procurement goals, particularly sustainability, within the procurement space.

Q: Nicole Darnall - Are you looking at innovative technology as it relates to sustainability and other aspects or just sustainability?
A: David Gill - My role goes beyond sustainability. I delve into various aspects of Federal procurement, including small businesses, category management, source selection, and contractor responsibility. My approach is shaped by my diverse experience, including roles as a supervisory IT specialist, a contracting officer representative, and involvement in the Chief Data Officer organization. Recently, I’ve also been active in academia, publishing papers through the Navy’s acquisition research symposium.

Q: Nicole Darnall - Tell us about your efforts to use artificial intelligence and technology to automate work within the IRS. What problems are you aiming to address, and who benefits from these solutions?
A: David Gill - I’ve focused on improving contract clauses to make them more accessible to the Acquisition Workforce. The Federal Acquisition Regulation and agency policies include around 1,000 clauses, making it overwhelming to select the relevant clauses for a given contract. Traditional contract writing systems were either unhelpful or overly burdensome, asking numerous questions. To simplify the process, I created a cloud application called the Contract Clause Review Tool, using natural language processing to analyze contract documents. It helps identify the appropriate clauses without overwhelming users. In 2021, our department prioritized sustainability, and I used the tool to align contracts with sustainability programs. For example, when I uploaded a managed print services solicitation document to the system software tool, it identified that I was purchasing printer toner, and showed me that USDA has it listed in their Biopreferred program. While there’s a proposed rule to consolidate sustainability into one clause, I still see value in using natural language processing tools for clarity and enforcement of sustainability measures, as they can help contracting officers understand and implement clauses effectively.

Q: Nicole Darnall - Can you explain the process regarding the printer toner and how the technology assists in this context? What problem does the technology solve in this case?
A: David Gill - In the case of a managed print services solicitation, I used my software to identify relevant clauses and review the document for errors or omissions. The technology helps streamline the process. For instance, it identified that FAR part 23 clauses for biobased products were missing, something the acquisition team had overlooked due to the complexity of biobased categories. The technology provided rapid recommendations, which would have otherwise required time-consuming research or coordination with experts. The speed of automation is a significant advantage in such situations.

Q: Nicole Darnall - Does the software contain the 1000+ clauses you mentioned? Is it comparing the solicitation content against these clauses, where the automation comes into play?
A: David Gill - Yes, and another key component is that it analyzes the words in the solicitation or contract document to understand what you are purchasing and detects specific attributes related to the procurement. For example, if the document mentions ‘Treasury headquarters,’ it should include the right clauses for that context. It can also identify attributes like biobased products, energy efficiency, or construction-related traits by analyzing keywords. The software’s ability to automatically detect these attributes is what saves the user time.

Q: Nicole Darnall - Considering Holly Elwood’s previous discussion on compliance issues with biobased and Energy Star procurement, do you think the system you’ve set up will potentially increase or reduce compliance in this space?
A: David Gill - The system is one tool that can help, but there are other tools and considerations in play. For example, the Small Business Administration has a scorecard for tracking small business utilization. In the realm of sustainability, compliance isn’t as straightforward. Contracting officers may lack expertise in sustainability and might find it challenging to determine the right steps for specific contracts. However, there’s potential to improve sustainability reporting. I’ve explored using artificial intelligence to validate free-form descriptions in the Federal Procurement Data System, which could enhancethe accuracy of sustainability reporting. Currently, agencies mostly focus on financial data validation, and sustainability is an area that hasn’t been scrutinized as much.

Q: Nicole Darnall - There are concerns about product or service codes not always being connected to sustainability attributes, making it hard to track and measure sustainability performance. Have you encountered this challenge in your work, or is it something you’ve addressed?
A: David Gill - It is something we’ve looked at. Product service codes were established in the 1970s and might not always align with modern technology or sustainability. While updating them could be beneficial, it’s complicated to select a single code for multifaceted areas like artificial intelligence, which plays a role in various government activities. Creating a specific code for AI might not reflect the contract’s primary purpose.

Nicole mentions a pain point related to the fact that many of the product codes are related to product families rather than individual products, causing sustainability data to be averaged across these families and making it challenging to pinpoint specific performance gains. David Gill offers to share slides related to an AI model for data validation that discusses these product service code families He discusses a challenge related to sustainability requirements in procurement, especially within the Treasury, where they lack a significant number of sustainability experts. The Federal Procurement Data System allows selecting sustainability attributes for contracts, but validating the data has been an issue. He highlights the need for accurate data, considering that training machine learning models with incorrect data can lead to inaccurate results. The challenge lies in contracts with free-form descriptions that don’t align with specific product service codes (PSC). David presents an AI model that leverages free-form contract descriptions to validate the selection of PSC codes and sustainability requirements.

The AI techniques used include text feature engineering and principal component analysis, which convert natural language into numbers while preserving word meanings. David suggests that combining the capabilities of GSA, such as AI expertise and sustainability knowledge, could create opportunities for technology reuse. He has data from the Federal Procurement Data System and USA Spending, aiming to review the suitability of selected PSC codes.

In his analysis, the AI model agreed with the contracting officers’ PSC code selection in most cases, but there was a 9% disagreement in a sample of 365,000 contracts, which could be a suitable population to audit for product data validation. Human review, particularly by those with sustainability expertise, can be helpful in addressing these discrepancies.

He also provides an example of a coding error, where the model recommended one category, but the contracting officer assigned a different one, illustrating the need for better alignment between codes and sustainability attributes.

Q: Kristin Seaver - What foundational elements are needed for implementing this system, and have you considered how to scale it across other agencies?
A: David Gill - Implementing this system requires certain infrastructure and specialized software, and having an AI specialist is essential, particularly when dealing with free-form descriptions and natural language. While it’s doable with technology, there are gaps in its integration into agency validation processes. This isn’t well-suited for shared services, as the output is a spreadsheet, indicating contract discrepancies. It’s useful for narrowing down the population for review. Scaling it across agencies would require a more systematic infrastructure, but currently, that isn’t in place.

David Gill shares a scorecard that provides an overview of sustainability levels in different agency contracts. He explains that the scorecard, based on data from 2021, was presented at the Federal Environment Symposium. He mentions that GSA has a dashboard website called D2D@gsa.gov (Data to Decisions) that offers a recurring updated scorecard to help agencies and contracting officers monitor their sustainability performance over time. This scorecard provides metrics and helps bring more attention to sustainability goals.

He proceeds to provide a timeline of his work, including the implementation of sustainability features in the Clause Tool in 2021. Gill emphasizes the importance of technology in helping contracting officials navigate the various requirements they deal with and introduces new sustainability clauses added to the tool. The software checks the Federal Acquisition Regulation (FAR) daily for new clauses. When a new clause is released, the software is designed to automatically identify it and incorporate it into their system. The software developers are proactive in keeping the system up to date, ensuring that they are informed about new or updated clauses in the FAR even before official policy updates are received through emails. This automation allows them to promptly include the latest sustainability clauses in their tool. He presents a chart showcasing policy and clause compliance within their documents, highlighting how the software helps users include the recommended clauses, leading to a win for sustainability whether a contract is marked in red or green.

Q: Nicole Darnall - If a clause is marked in orange, is that where we need validation or follow-up?
A: David Gill - The orange indicates where the software is not entirely accurate. Sometimes, the software makes educated guesses based on keywords and context. However, there are cases in the FAR regulations where specific instructions state not to apply certain clauses to small businesses or certain types of contracts. These clauses might be considered overly burdensome for those cases. So, when the software marks a clause in orange, it signifies that it’s potentially including an applicable requirement in the contract that shouldn’t be there, considering the government’s policy stance on certain contract types. It’s like a caution for further evaluation.
David Gill continues with the powerpoint presentation.

Our collaboration with academia involves reading research papers, conducting workforce training, and sharing insights. We have particularly focused on sustainability expertise, leveraging the help of Leo Gumapas from Health and Human Services. This collaboration includes analyzing federal data, specifically related to contract spending. The Federal Procurement Data System serves as a standardized source of data, ensuring consistency across agencies.

One research finding draws parallels with behavioral science, likening it to product placement in stores, where strategically positioning products affects visibility. We aim to make sustainability resources more visible to procurement professionals. This involves looking beyond the “choir” of sustainability experts and making these resources more accessible to the broader procurement workforce.

In summary, there are several recommendations to enhance sustainability in federal procurement:

• Emphasize Websites: Place a strong emphasis on websites that the Acquisition Workforce frequently accesses, ensuring that sustainability resources are prominently featured.
• Product or Service Code Validation: Verify that a reasonable PSC code was selected which helps to validate the application of sustainability requirements in contracts.
• Sustainable Procurement Scorecard: Involves creating a scorecard or dashboard for tracking and monitoring sustainability performance.
• Utilize Natural Language Processing: Leverage natural language processing technology to gain deeper insights into contract documents, as these documents are central to the procurement process.
• Green Procurement Audits: Consider adopting green procurement audits, a practice demonstrated in certain parts of the Department of Defense, such as the Army. Auditing can help ensure compliance with sustainability requirements and add enforcement mechanisms to clauses and dashboards.

Q: Nicole Darnall - Could you discuss the challenges or blockers you encountered while exploring automation opportunities within IRS & Treasury and how these might be relevant to our Subcommittee?
A: David Gill - Privacy considerations have generally gone smoothly for us. I conducted a privacy impact assessment, and most agencies have issued a system of record notices regarding their procurement data, authorizing its use for sustainability purposes. We don’t handle personal identifiable information in this context, but engaging with privacy and documenting the process is essential. Cybersecurity can be more challenging, especially with advanced technologies like natural language processing.

For instance, when building AI technology, cybersecurity can become complex, and thorough reviews are needed. Another challenge is the limited expertise and understanding among the Acquisition Workforce. To overcome this, a human-centered design approach and user experience are crucial. Our Clause Tool software, for instance, emerged from my experiences as a contracting officer, and user feedback has been invaluable in improving it.

Q: Nicole Darnall - Did you partner with an AI firm to help develop these applications?
A: David Gill - I partnered with AI firms in some cases, but I developed the sustainability AI model myself. We also purchased commercial AI software. It simplifies tasks like converting freeform descriptions into an AI-understandable format, which can be challenging to do through code alone. This kind of software lowers the barriers to entry for individuals with some knowledge of AI and can be used by citizen data scientists to solve various problems.

Q: Kristin Seaver - How can we get started with this concept? Even on a small scale, are there common misconceptions or common mistakes that we can learn from and share with the community?
A: David Gill - Consider the Federal Acquisition Institute at GSA. There’s sustainability training available there, but sharing common mistakes, lessons learned, and misunderstood concepts could be beneficial. Providing concrete examples can be more engaging and help the community understand the importance.

Q: Nicole Darnall - If GSA were to launch a pilot to explore this broader opportunity space, where do you think would be the best place to start based on your knowledge of procurement?
A: David Gill - IT systems acquisition offers significant opportunities, mainly due to the lack of guidance. While purchasing sustainable laptops or managing energy-efficient building renovations is straightforward, IT systems involve facilities, servers, and software. Evaluating and selecting the most sustainable IT systems vendor, surveying their sustainability, managing server-related costs, and ensuring energy efficiency in AI algorithms are emerging areas. GSA could develop evaluation tools and templates that benefit both GSA and other agencies.

Q: Nicole Darnall - What are your plans for publishing and disseminating your research to a wider audience?
A: David Gill - We plan to have it open to the public, but currently we are still in the process of agency review.

Q: Stephanie Hardison - Was there any research in the cost comparison between purchasing commercial systems versus building your own?
A: David Gill - In the case of natural language processing, I buy a commercial software product through a small business. The Department of Homeland Security has used the same firm, but there are also other private-sector vendors specializing in natural language and AI software. The AI model for procurement data validation developed by the government is not proprietary to a specific vendor and doesn’t involve intellectual property rights. It’s a government product. I’ve used both commercial and
government-developed solutions, depending on what is the most efficient path to achieve the goal.

Q: Nicole Darnall - Would it be possible to follow-up with you to get information on specific names of the off-the-shelf products that have been utilized in different ways?
A: David Gill - Yes, I have a market research report that will name vendor names.

Q: Nicole Darnall - Are there immediate policy changes or recommendations that you would advocate for to advance sustainability in Federal acquisition?
A: David Gill - While I wouldn’t recommend regulatory changes, there are things that GSA could consider internally. One potential change is amending the FPDS data validation and verification policy to include sustainable procurement data validation. Additionally, incorporating emerging technologies like automation and AI into the planning activities can broaden the range of available procurement technology options.

Q: Nicole Darnall - You mentioned the GSA Centers of Excellence. Is there an AI focus within those centers?
A: David Gill - Yes, there is at least one group within GSA, and likely multiple groups, that have expertise in AI. Having AI expertise is crucial to effectively run AI models. If I provide source codes and data for an AI model to someone who isn’t familiar with hands-on AI data, they may struggle. An AI expert would have the experience and infrastructure to run AI models. In this case, the output would be a spreadsheet for use in quarterly data validation and verification processes.

Q: Nicole Darnall - Is there anything else you’d like to add based on your experience and the opportunities ahead for GSA in terms of automation and streamlining processes?
A: David Gill - Acquisition Gateway has a tool called GSA eBuy, which serves as a portal for sending requests for quotes or solicitations to GSA vendors. It has an open API to the document library in eBuy, allowing users to access GSA solicitations. This could be valuable for examining sustainability requirements within those solicitations. APIs are machine-readable, providing opportunities for AI to analyze documents for things like sustainability clauses. A pilot project within an existing system like this could be a promising way to explore these possibilities, focusing on a specific subset of procurements. This leverages existing technology infrastructure and resources.

Q: Nicole Darnall - To approach a potential pilot, would it be practical to focus on a specific category like IT services or a particular role within GSA, such as the Acquisition Gateway?
A: David Gill - You need a system as a foundation for implementing AI or automation. You can’t build these technologies from scratch. The Acquisition Gateway is one such system, which includes solicitation documents. While it doesn’t cover every Federal or GSA solicitation, it covers a subset. From what I understand about the technology stack, the fact that there is an application programming interface (API) to access these solicitation documents is significant. This API could potentially facilitate communication between AI servers, such as those in the Technology Transformation Service, and the servers where e-Buy data is stored.

Q: Darryl Daniels - You’ve mentioned that small business procurement metrics were simpler and more advanced compared to sustainability procurement. Is it possible for those who specialize in small business procurement to share their insights with the sustainability procurement process, considering the unique certification and regulatory aspects of the small business world?
A: David Gill - We do have small business government advocates embedded in our procurement process who work with contracting officers. They help us navigate the complexities of small business regulations and certifications. These advocates are experts in this field. Similarly, for sustainability, we have policy and sustainability experts who can provide guidance on regulations and questions. However, having specialists who actively advocate for sustainability in the procurement process is a different approach. It could be beneficial to integrate advocacy and reviews for sustainability into the acquisition process, similar to how small business advocates influence decisions.

This could help contracting officers, who may not be sustainability experts, consider energy efficiency and other sustainability requirements in their acquisitions more effectively.

The group thanked David Gill for his time and presentation to the subcommittee.

Review of key takeaways

Nicole opened the Jamboard for the subcommittee to reflect on the points and takeaways they heard from the guest speaker. The group came up with the following:

• Sustainability advocacy.
• Scorecards exist for small business utilization which are graded. The same should be done for sustainability.
• Sentiment: as a buyer you are a generalist, emerging requirements in cyber, climate, SB, etc, bring in a lot of complexity.
• Manage human error.
• Category management is consolidated. Can see what percent of spend is consistent with expectations. Do the same in category management for sustainability.
• Does GSA need a Chief Data Officer and an Office of Data Management?
• Loved hearing about the competition. This flows over into IPS work. We could include something in our pipeline that would also help AWFS.
• Can learn from the output from even a small percent of use cases as to what is commonly seen.
• A beginning pathway provide (maybe): AI specialists, technology, domain injects and then gave us some good organizations to talk to within GSA FAS
• The scanning of product words and then the lookup of key federal procurement information and key environmental programs.
• IRS has an analytics, research and technology division.
• Privacy is less of a concern than cyber security.
• Sustainability review process is needed - similar to small business review (with advocacy process and scorecard)
• Consider a GSA pilot in IT systems acquisition, especially the Acquisition Gateway and E-buy as a starting place.
• Sustainability advocacy.
• Task force for AI?
• Need FPDS policy verification and validation - amend to include sustainable procurement data validation, just like financial data validation.
• Tap into the expertise of the GSA AI team.
• AI could and should be used for procurement services. There are so many opportunities to automate.
• Green procurement audits - DTIC (Army); audit what contractor is delivering.

Nicole suggests the possibility of creating a task force on AI within GSA to explore internal opportunities and dedicate expertise to address technology-related issues, especially in procurement. She acknowledges David Gill’s efforts in identifying the potential of AI in automating procurement and highlights the significance of 91% contract validation accuracy achieved through AI. She envisions reallocating resources to improve the remaining 9% of AI performance to accelerate sustainability efforts within the organization.

Kristin Seaver suggests creating a pipeline of challenges as part of the Industry Partnerships subcommittee’s recommendations, which could support the Acquisition Workforce Subcommittee work. She proposes including challenges related to AI in contract and procurement as an example, gathering information already available in the field. Nicole agreed with the idea.

Nicole discussed the need for centralized expertise and more information before moving forward with recommendations related to technology tools and AI in procurement.

Kristin suggests creating challenges related to AI in procurement to gather information. They discuss the complexity of procurement and the potential of AI tools to assist professionals.

Stephanie Hardison emphasized the importance of involving the Center of Excellence. It will give the group a sense of direction of where GSA is going with AI and procurement, and will give them more insight into where they should take it.

Public engagement

Nicole Darnall opened the discussion up to the public, but there were no questions or comments.

Closing remarks

Nicole Darnall thanked the subcommittee for their participation. She reiterated that her and Anne will be compiling a draft recommendation report to share with the subcommittee. The first recommendation is related to third-party training and what optimal third-party training would look like. The second area relates to experiential learning. It is something that the subcommittee believed to be an effective form of learning. The group will also need to ask themselves what elements of experiential learning do they think are most important for GSA to consider? The third priority area for recommendations was looking at the data standardization and developing training on existing data tools and systems.

In the next suite of recommendations after the December meeting, the group will focus on AI and also on the notion of internal incentives within the Acquisition Workforce.

How do you reward individuals that are going the extra mile, and create internal incentives for individuals to wrestle with this inherently complex field of managing sustainability alongside all of these other expectations?

Adjournment

Stephanie Hardison adjourned the meeting at 5:00 P.M. EST.

I hereby certify that, to the best of my knowledge, the foregoing minutes are accurate and complete.

Digitally signed by Nicole Darnall 3/25/2024
Nicole Darnall
Chairperson
GAP FAC Acquisition Workforce Subcommittee

Digitally signed by Anne Rung 3/25/2024
Anne Rung
Co-Chairperson
GAP FAC Acquisition Workforce Subcommittee