Jump to content

User:CaveatLector2022/Tests

From Wikipedia, the free encyclopedia

Financial markets

[edit]

The AI technology company c3.ai saw a 28% increase in its share price after announcing the integration of ChatGPT into its toolkit.[1] The share price of BuzzFeed, a digital media company unrelated to AI, increased 120% after announcing OpenAI technology adoption for content creation.[2] Reuters found that share prices of AI-related companies BigBear.ai and SoundHound AI increased by 21% and 40%, respectively, even though they had no direct connection to ChatGPT.[3] They attributed this surge to ChatGPT's role in turning AI into Wall Street's buzzword. Academic research published in Finance Research Letters found that the 'ChatGPT effect' prompted retail investors to drive up prices of AI-related cryptocurrency assets despite the broader cryptocurrency market being in a bear market, and diminished institutional investor interest.[4] This confirms anecdotal findings by Bloomberg that, in response to ChatGPT's launch, cryptocurrency investors showed a preference for AI-related crypto assets.[5] An experiment by finder.com revealed that ChatGPT could outperform popular fund managers by picking stocks based on criteria such as growth history and debt levels, resulting in a 4.9% increase in a hypothetical account of 38 stocks, outperforming 10 benchmarked investment funds with an average loss of 0.8%.[6]

Conversely, executives and investment managers at Wall Street quant funds (including those that have used machine learning for decades) have noted that ChatGPT regularly makes obvious errors that would be financially costly to investors because even AI systems that employ reinforcement learning or self-learning have had only limited success in predicting market trends due to the inherently noisy quality of market data and financial signals.[7] In November 2023, research conducted by Patronus AI, an artificial intelligence startup company, compared performance of GPT-4, GPT-4-Turbo, Claude2, and LLaMA-2 on two versions of a 150-question test about information in financial statements (e.g. Form 10-K, Form 10-Q, Form 8-K, earnings reports, earnings call transcripts) submitted by public companies to the U.S. Securities and Exchange Commission. One version of the test required the generative AI models to use a retrieval system to find the specific SEC filing to answer the questions; the other gave the models the specific SEC filing to answer the question (i.e. in a long context window). On the retrieval system version, GPT-4-Turbo and LLaMA-2 both failed to produce correct answers to 81% of the questions, while on the long context window version, GPT-4-Turbo and Claude-2 failed to produce correct answers to 21% and 24% of the questions, respectively.[8][9]

Medicine

[edit]
See also: Artificial intelligence in healthcare

In the field of health care, possible uses and concerns are under scrutiny by professional associations and practitioners.[10][11] Two early papers indicated that ChatGPT could pass the United States Medical Licensing Examination (USMLE).[12] MedPage Today noted in January 2023 that "researchers have published several papers now touting these AI programs as useful tools in medical education, research, and even clinical decision making."[12]

Published in February 2023 were two separate papers that again evaluated ChatGPT's proficiency in medicine using the USMLE. Findings were published in JMIR Medical Education (see Journal of Medical Internet Research) and PLOS Digital Health. The authors of the PLOS Digital Health paper stated that the results "suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making."[13][14] In JMIR Medical Education, the authors of the other paper concluded that "ChatGPT performs at a level expected of a third-year medical student on the assessment of the primary competency of medical knowledge." They suggest that it could be used as an "interactive learning environment for students". The AI itself, prompted by the researchers, concluded that "this study suggests that ChatGPT has the potential to be used as a virtual medical tutor, but more research is needed to further assess its performance and usability in this context."[15] The later-released ChatGPT version based on GPT-4 significantly outperformed the version based on GPT-3.5.[16] Researchers at Stanford University and the University of California, Berkeley have found that the performance of GPT-3.5 and GPT-4 on the USMLE declined from March 2023 to June 2023.[17][18]

A March 2023 paper tested ChatGPT's application in clinical toxicology. The authors found that the AI "fared well" in answering a "very straightforward [clinical case example], unlikely to be missed by any practitioner in the field". They added: "As ChatGPT becomes further developed and specifically adapted for medicine, it could one day be useful in less common clinical cases (i.e, cases that experts sometimes miss). Rather than AI replacing humans (clinicians), we see it as 'clinicians using AI' replacing 'clinicians who do not use AI' in the coming years."[19]

An April 2023 study in Radiology tested the AI's ability to answer queries about breast cancer screening. The authors found that it answered appropriately "about 88 percent of the time", however, in one case (for example), it gave advice that had become outdated about a year earlier. The comprehensiveness of its answers was also lacking.[20][21] A study published in JAMA Internal Medicine that same month found that ChatGPT often outperformed human doctors at answering patient questions (when measured against questions and answers found at /r/AskDocs, a forum on Reddit where moderators validate the medical credentials of professionals; the study acknowledges the source as a limitation).[22][23][24] The study authors suggest that the tool could be integrated with medical systems to help doctors draft responses to patient questions.[25][26]

Professionals have emphasized ChatGPT's limitations in providing medical assistance. In correspondence to The Lancet Infectious Diseases, three antimicrobial experts wrote that "the largest barriers to the implementation of ChatGPT in clinical practice are deficits in situational awareness, inference, and consistency. These shortcomings could endanger patient safety."[27] Physician's Weekly, though also discussing the potential use of ChatGPT in medical contexts (e.g. "as a digital assistant to physicians by performing various administrative functions like gathering patient record information or categorizing patient data by family history, symptoms, lab results, possible allergies, et cetera"), warned that the AI might sometimes provide fabricated or biased information.[28] One radiologist warned: "We've seen in our experience that ChatGPT sometimes makes up fake journal articles or health consortiums to support its claims";[29] As reported in one Mayo Clinic Proceedings: Digital Health paper, ChatGPT may do this for as much as 69% of its cited medical references. The researchers emphasized that while many of its references were fabricated, those that were appeared "deceptively real".[30] As Dr. Stephen Hughes mentioned for The Conversation however, ChatGPT is capable of learning to correct its past mistakes. He also noted the AI's "prudishness" regarding sexual health topics.[31]

Contrary to previous findings, ChatGPT responses to anesthesia-related questions were more accurate, succinct, and descriptive compared to Bard's. Bard exhibited 30.3% error in response as compared to ChatGPT (0% error).[32] At a conference of the American Society of Health-System Pharmacists in December 2023, researchers at Long Island University (LIU) presented a study that researched ChatGPT's responses to 45 frequently asked questions of LIU College of Pharmacy's drug information service during a 16-month period from 2022 to 2023 as compared with researched responses provided by professional pharmacists. For 29 of the 39 questions for which there was sufficient medical literature for a data-driven response, ChatGPT failed to provide a direct answer or provided a wrong or incomplete answer (and in some cases, if acted upon, the answer would endanger the patient's health). The researchers had asked ChatGPT to provide medical research citations for all its answers, but it did so for only eight, and all eight included at least one fabricated (fake) citation.[33][34]

A January 2024 study conducted by researchers at Cohen Children's Medical Center found that GPT-4 had an accuracy rate of 17% when diagnosing pediatric medical cases.[35][36]

Law

[edit]

In January 2023, Massachusetts State Senator Barry Finegold and State Representative Josh S. Cutler proposed a bill partially written by ChatGPT, "An Act drafted with the help of ChatGPT to regulate generative artificial intelligence models like ChatGPT",[37][38][39] which would require companies to disclose their algorithms and data collection practices to the office of the State Attorney General, arrange regular risk assessments, and contribute to the prevention of plagiarism.[38][39][40] The bill was officially presented during a hearing on July 13.[37][39]

On April 11, 2023, a judge of a session court in Pakistan used ChatGPT to decide the bail of a 13-year-old accused in a matter. The court quoted the use of ChatGPT assistance in its verdict:

The AI language model replied:

The judge asked ChatGPT other questions about the case and formulated his final decision in light of its answers.[41][42]

In Mata v. Avianca, Inc., 22-cv-1461 (PKC), a personal injury lawsuit against Avianca Airlines filed in the Southern New York U.S. District Court in May 2023 (with Senior Judge P. Kevin Castel presiding), the plaintiff's attorneys reportedly used ChatGPT to generate a legal motion. ChatGPT generated numerous fictitious legal cases involving fictitious airlines with fabricated quotations and internal citations in the legal motion. Castel noted numerous inconsistencies in the opinion summaries, and called one of the cases' legal analysis "gibberish".[43] The plaintiff's attorneys faced potential judicial sanction and disbarment for filing the motion and presenting the fictitious legal decisions ChatGPT generated as authentic.[44][45] The case was dismissed and the attorneys were fined $5,000.[46][47]

In October 2023, the council of Porto Alegre, Brazil, unanimously approved a local ordinance proposed by councilman Ramiro Rosário that would exempt residents from needing to pay for the replacement of stolen water consumption meters; the bill went into effect on November 23. On November 29, Rosário revealed that the bill had been entirely written by ChatGPT, and that he had presented it to the rest of the council without making any changes or disclosing the chatbot's involvement.[40][48][49] The city's council president, Hamilton Sossmeier, initially criticized Rosário's initiative, saying it could represent "a dangerous precedent",[49][50] but later said he "changed his mind": "unfortunately or fortunately, this is going to be a trend."[40]

  1. ^ Fox, Matthew (January 31, 2023). "C3ai has soared 86% year-to-date as investor frenzy for artificial intelligence builds amid ChatGPT success". Markets Insider (Business Insider). Archived from the original on February 18, 2023. Retrieved April 14, 2023.
  2. ^ Diaz, Alicia; Smith, Gerry (January 26, 2023). "BuzzFeed Shares Surge 120% on Plans to Embrace OpenAI". Bloomberg.com. Retrieved May 22, 2023.
  3. ^ Singh, Medha; Biswas, Ankika (February 6, 2023). "AI stocks rally in latest Wall Street craze sparked by ChatGPT". Reuters. Archived from the original on March 29, 2023. Retrieved April 14, 2023.
  4. ^ Saggu, Aman; Ante, Lennart (May 8, 2023). "The influence of ChatGPT on artificial intelligence-related crypto assets: Evidence from a synthetic control analysis". Finance Research Letters. 55: 103993. arXiv:2305.12739. doi:10.1016/j.frl.2023.103993. ISSN 1544-6123. S2CID 258573881.
  5. ^ Hajric, Vildana; Shen, Muyao (February 9, 2023). "ChatGPT Mania Spurs Crypto Fans' Stampede to 'Faddish' AI Tokens". Bloomberg.com. Archived from the original on February 9, 2023. Retrieved April 14, 2023.
  6. ^ Cooban, Anna (May 5, 2023). "ChatGPT can pick stocks better than your fund manager". CNN. Archived from the original on May 22, 2023. Retrieved May 5, 2023.
  7. ^ Zuckerman, Gregory (April 12, 2023). "AI Can Write a Song, but It Can't Beat the Market". The Wall Street Journal. News Corp. Archived from the original on May 30, 2023. Retrieved May 30, 2023.
  8. ^ Leswing, Kif (December 19, 2023). "GPT and other AI models can't analyze an SEC filing, researchers find". CNBC. Archived from the original on December 19, 2023. Retrieved December 19, 2023.
  9. ^ "Patronus AI Launches Industry-first LLM Benchmark for Finance to Address Hallucinations" (Press release). PR Newswire. November 16, 2023. Archived from the original on December 19, 2023. Retrieved December 19, 2023.
  10. ^ The Lancet Digital Health (March 3, 2023). "ChatGPT: friend or foe?". The Lancet Digital Health. 5 (3): e102. doi:10.1016/S2589-7500(23)00023-7. PMID 36754723. S2CID 256659547.
  11. ^ Asch, David A. (April 4, 2023). "An Interview with ChatGPT About Health Care". NEJM Catalyst Innovations in Care Delivery. doi:10.1056/CAT.23.0043 (inactive January 31, 2024). Archived from the original on June 29, 2023. Retrieved June 29, 2023.{{cite journal}}: CS1 maint: DOI inactive as of January 2024 (link)
  12. ^ a b DePeau-Wilson, Michael (January 19, 2023). "AI Passes U.S. Medical Licensing Exam". MedPage Today. Archived from the original on April 9, 2023. Retrieved May 2, 2023.
  13. ^ Kung, Tiffany H.; Cheatham, Morgan; Medenilla, Arielle; Sillos, Czarina; Leon, Lorie De; Elepaño, Camille; Madriaga, Maria; Aggabao, Rimel; Diaz-Candido, Giezel; Maningo, James; Tseng, Victor (February 9, 2023). "Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models". PLOS Digital Health. 2 (2): e0000198. doi:10.1371/journal.pdig.0000198. ISSN 2767-3170. PMC 9931230. PMID 36812645.
  14. ^ "Expert reaction to study on ChatGPT almost passing the US Medical Licensing Exam". Science Media Centre. February 9, 2023. Archived from the original on April 24, 2023. Retrieved May 2, 2023.
  15. ^ Gilson, Aidan; Safranek, Conrad W.; Huang, Thomas; Socrates, Vimig; Chi, Ling; Taylor, Richard Andrew; Chartash, David (February 8, 2023). "How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment". JMIR Medical Education. 9 (1): e45312. doi:10.2196/45312. PMC 9947764. PMID 36753318.
  16. ^ Brueck, Hilary. "The newest version of ChatGPT passed the US medical licensing exam with flying colors — and diagnosed a 1 in 100,000 condition in seconds". Business Insider. Archived from the original on January 27, 2024. Retrieved February 2, 2024.
  17. ^ Pressman, Aaron (November 8, 2023). "The AI boom is shaking up the tech industry and moving markets. But is it all a mirage?". The Boston Globe. Archived from the original on December 5, 2023. Retrieved December 5, 2023.
  18. ^ Chen, Lingjiao; Zaharia, Matei; Zou, James (October 31, 2023). "How is ChatGPT's behavior changing over time?". arXiv:2307.09009v3 [cs.CL].
  19. ^ Abdel-Messih, Mary Sabry; Boulos, Maged N. Kamel (March 8, 2023). "ChatGPT in Clinical Toxicology". JMIR Medical Education. 9 (1): e46876. doi:10.2196/46876. PMC 10034604. PMID 36867743.
  20. ^ Haver, Hana L; Ambinder, Emily B; Bahl, Manisha; Oluyemi, Eniola T; Jeudy, Jean; Yi, Paul H (April 4, 2023). "Appropriateness of Breast Cancer Prevention and Screening Recommendations Provided by ChatGPT". Radiology. 307 (4): 230424. doi:10.1148/radiol.230424. ISSN 0033-8419. PMID 37014239. S2CID 257923990. Archived from the original on May 5, 2023. Retrieved May 5, 2023.
  21. ^ Kotz, Deborah (April 4, 2023). "UM School of Medicine Study Finds ChatGPT Helpful for Breast Cancer Screening Advice, With Certain Caveats". University of Maryland School of Medicine. Archived from the original on May 5, 2023. Retrieved May 5, 2023.
  22. ^ Ayers, John W.; Poliak, Adam; Dredze, Mark; Leas, Eric C.; Zhu, Zechariah; Kelley, Jessica B.; Faix, Dennis J.; Goodman, Aaron M.; Longhurst, Christopher A.; Hogarth, Michael; Smith, Davey M. (April 28, 2023). "Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum". JAMA Internal Medicine. 183 (6): 589–596. doi:10.1001/jamainternmed.2023.1838. ISSN 2168-6106. PMC 10148230. PMID 37115527. Archived from the original on April 30, 2023. Retrieved May 2, 2023.{{cite journal}}: CS1 maint: PMC embargo expired (link)
  23. ^ Fox, Andrea (May 4, 2023). "Does ChatGPT really outshine doctors? Or just on social media?". Healthcare IT News. HIMSS Media. Archived from the original on May 4, 2023. Retrieved May 5, 2023.
  24. ^ "The doctor is out, but it's OK. ChatGPT can answer your questions". Hub. Johns Hopkins University. April 28, 2023. Archived from the original on May 5, 2023. Retrieved May 5, 2023.
  25. ^ Ono, Mika (April 28, 2023). "Study Finds ChatGPT Outperforms Physicians in High-Quality, Empathetic Answers to Patient Questions". UC San Diego Today. Archived from the original on April 28, 2023. Retrieved April 28, 2023.
  26. ^ "ChatGPT Beats Doctors in Compassion and Quality of Advice to Patients". Neuroscience News. April 28, 2023. Archived from the original on May 3, 2023. Retrieved May 2, 2023.
  27. ^ Howard, Alex; Hope, William; Gerada, Alessandro (April 2023). "ChatGPT and antimicrobial advice: the end of the consulting infection doctor?". The Lancet Infectious Diseases. 23 (4): 405–406. doi:10.1016/s1473-3099(23)00113-5. ISSN 1473-3099. PMID 36822213. S2CID 257072872. Archived from the original on March 25, 2023. Retrieved May 2, 2023.
  28. ^ "Is There a Role for ChatGPT in Healthcare?". Physician's Weekly. April 27, 2023. Archived from the original on May 5, 2023. Retrieved May 5, 2023.
  29. ^ Drake, Kimberly (April 6, 2023). "Rely on a Doctor, Not ChatGPT, for Medical Advice". HealthNews. Archived from the original on May 5, 2023. Retrieved May 5, 2023.
  30. ^ Gravel, Jocelyn; D’Amours-Gravel, Madeleine; Osmanlliu, Esli (September 1, 2023). "Learning to Fake It: Limited Responses and Fabricated References Provided by ChatGPT for Medical Questions". Mayo Clinic Proceedings: Digital Health. 1 (3): 226–234. doi:10.1016/j.mcpdig.2023.05.004. ISSN 2949-7612.
  31. ^ Hughes, Stephen (April 27, 2023). "How good is ChatGPT at diagnosing disease? A doctor puts it through its paces". The Conversation. Archived from the original on May 4, 2023. Retrieved May 5, 2023.
  32. ^ Patnaik, Sourav S.; Hoffmann, Ulrike (November 7, 2023). "Quantitative evaluation of ChatGPT versus Bard responses to anaesthesia-related queries". British Journal of Anaesthesia. 132 (1): S0007–0912(23)00550–0. doi:10.1016/j.bja.2023.09.030. ISSN 1471-6771. PMID 37945414. S2CID 265078930. Archived from the original on February 18, 2024. Retrieved November 28, 2023.
  33. ^ Constantino, Annika Kim (December 5, 2023). "Free ChatGPT may incorrectly answer drug questions, study says". CNBC. Archived from the original on December 5, 2023. Retrieved December 5, 2023.
  34. ^ "Study Finds ChatGPT Provides Inaccurate Responses to Drug Questions" (Press release). PR Newswire. December 5, 2023. Archived from the original on December 5, 2023. Retrieved December 5, 2023.
  35. ^ Barile, Joseph; Margolis, Alex; Cason, Grace; Kim, Rachel; Kalash, Saia; Tchaconas, Alexis; Milanaik, Ruth (January 2, 2024). "Diagnostic Accuracy of a Large Language Model in Pediatric Case Studies". JAMA Pediatrics. 178 (3): 313–315. doi:10.1001/jamapediatrics.2023.5750. ISSN 2168-6203. PMC 10762631. PMID 38165685. Archived from the original on February 18, 2024. Retrieved February 18, 2024.{{cite journal}}: CS1 maint: PMC embargo expired (link)
  36. ^ Mole, Beth (January 3, 2024). "ChatGPT bombs test on diagnosing kids' medical cases with 83% error rate". Ars Technica. Archived from the original on January 17, 2024. Retrieved January 5, 2024.
  37. ^ a b "Bill S.31". malegislature.gov. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
  38. ^ a b Annear, Steve (January 24, 2023). "Two elected officials drafted legislation to regulate artificial intelligence technology — with some help from ChatGPT". The Boston Globe. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
  39. ^ a b c Garrity, Kelly; Kashinsky, Lisa (July 13, 2023). "ChatGPT enters the legislative chat". POLITICO. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
  40. ^ a b c Quach, Katyanna (December 2, 2023). "Local council in Brazil passes ChatGPT-written proposal". The Register. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
  41. ^ "Pakistani judge uses ChatGPT to make court decision". Gulf News. April 13, 2023. Archived from the original on April 20, 2023. Retrieved April 20, 2023.
  42. ^ "AI revolution is here': Pakistani court takes help from ChatGPT to grant bail in rape case". Pakistan Observer. April 11, 2023. Archived from the original on April 20, 2023. Retrieved April 20, 2023.
  43. ^ Brodkin, Jon (June 23, 2023). "Lawyers have real bad day in court after citing fake cases made up by ChatGPT". Ars Technica. Archived from the original on January 26, 2024. Retrieved February 18, 2024.
  44. ^ Goswami, Rohan (May 30, 2023). "ChatGPT cited 'bogus' cases for a New York federal court filing. The attorneys involved may face sanctions". CNBC. Archived from the original on May 30, 2023. Retrieved May 30, 2023.
  45. ^ Neumeister, Larry (June 8, 2023). "Lawyers blame ChatGPT for tricking them into citing bogus case law". Associated Press. Archived from the original on November 8, 2023. Retrieved November 8, 2023.
  46. ^ "Mata v. Avianca, Inc". Casetext. Archived from the original on February 13, 2024. Retrieved February 13, 2024.
  47. ^ "'Use with caution': How ChatGPT landed this US lawyer and his firm in hot water". ABC News. June 24, 2023. Archived from the original on November 9, 2023. Retrieved November 9, 2023.
  48. ^ Jeantet, Diane; Savarese, Mauricio; LeBlanc, Steve; O'Brien, Matt (November 30, 2023). "Brazilian city enacts an ordinance that was secretly written by ChatGPT". AP News. Associated Press. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
  49. ^ a b Paúl, María Luisa (December 4, 2023). "A Brazilian city passed a law about water meters. ChatGPT wrote it". Washington Post. ISSN 0190-8286. Archived from the original on December 5, 2023. Retrieved December 7, 2023.
  50. ^ "Lei escrita por inteligência artificial é aprovada por vereadores em Porto Alegre; 'precedente perigoso', diz presidente da Câmara". G1 (in Brazilian Portuguese). November 29, 2023. Archived from the original on December 7, 2023. Retrieved December 7, 2023.
Retrieved from "https://en.wikipedia.org/w/index.php?title=User:CaveatLector2022/Tests&oldid=1212812635"