Management Matters: The value and limits of AI

Jeffrey R. Schoenberger is senior consultant with Affinity Consulting Group, LLC. 

The value and limits of AI

No matter where you turn today, you read about the opportunities -- and -- perils of artificial intelligence (AI) tools like ChatGPT. Instead of focusing on the extremes, let’s focus on the banalities that legal professionals need to know when approaching AI.

A writing aid, but not (yet) a writer

One productive use of AI is to overcome writer’s block. Because ChatGPT works like a conversation, it’s possible to ask questions to help get your own ideas flowing. For example, when asked, “What are the most popular Outlook 365 features?” for a presentation, ChatGPT instantly returned a list of nine items, with a sentence describing each.

This list may spur inspiration or break a mental logjam, but even this simple request exposes gaps. First, its top four items are, in order, “email management,” “calendar integration,” “contacts and address book,” and “task management.” True enough, but it hardly makes for an illuminating webinar to say, “Outlook does email.”

Second, AI tools like ChatGPT are known as large language models (LLM). Companies train these LLMs with an initial data set. ChatGPT and other AIs have cutoffs after which they add no new data; in ChatGPT's case, the cutoff is September 2021. So, were I to ask, “What new features of Outlook 365 should lawyers know about?”, it would be ignorant of features added after September 2021.

Third, ChatGPT is verbose. When prompted: “Write a 500-word blog about the best use of Outlook by lawyers,” it supplied an article of 600 words. It’s impressively grammatically correct but could benefit from editing.

Create an account and try to break your writer’s block here: Chat.OpenAI.com.

AI tries hard to please

On June 22, 2023, Judge Kevin Castel of New York’s Southern District fined lawyers Steven Schwartz and Pete LoDuca $5,000 for filing a brief written by ChatGPT. Unfortunately for the lawyers involved, ChatGPT didn’t merely engage in the above-mentioned foibles. Instead, it invented cases and holdings to fit the lawyers' propositions. Furthermore, ChatGPT’s manufactured cases included properly formatted --but entirely fake -- citations, and it attributed the holdings to real judges. As part of his sanctions ruling, Castel ordered the lawyers to send letters to “the judges whose names were wrongfully invoked,” notifying them of the sanctions.

Lawyers sometimes feel rushed to get work out the door, meet a filing deadline, or check off a drafting task before vacation. And while ChatGPT and other generative-AI tools may help overcome writer’s block, legal professionals must review everything AI supplies, particularly for pleadings and work products.

To experiment with legal industry-targeted AI tools, try Casetext’s CoCounsel, Lexis + AI, or Westlaw Edge.

Two worlds of AI

Stephen Rose, chief technology strategist at Petri IT, was interviewed on the First Ring Daily podcast about Microsoft’s AI announcements at its 2023 Build Conference. He offered a statistic from Microsoft’s Work Trend Index: 70% of employees would hand off as much work as possible to AI.

Additional numbers Rose cites are perhaps even more alarming: 70% of respondents are experimenting with ChatGPT -- and40% of those experimenting are putting confidential information into it. ChatGPT and most public-facing AI tools store the questions they’re asked. If you or someone in your firm is entering confidential information into an AI tool, stop until you know what the vendor does with the questions asked of the AI.

Rose’s suggested solution for confidentiality is to rely on a “sandboxed” AI rather than a “general public” tool. For businesses, and perhaps larger firms at present, that entails using AI tools available through Microsoft, Amazon, or others, running in a cloud environment that the business controls. Then the firm could train or “seed” its AI sandbox with the documents it stores in SharePoint while having user permissions controlled through Active Directory. The firm maintains information security while teaching an AI to “think and write” like the firm We’re on the way to where a “best of breed” AI can combine reference material and case law with your past work product to create a quality draft in record time.

But you should still proof it before filing.