PT study finds the Dell PowerEdge XE9680 servers with NVIDIA H100 GPUs supports many chatbot users
Running your in-house chatbot using Llama 3.1 405B LLMs on Dell PowerEdge XE9680 servers with NVIDIA H100 GPUs
According to the report, Principled Technologies “used the PTChatterly service to showcase the AI chatbot performance of a Dell™ PowerEdge XE9680 server powered by eight NVIDIA H100 SXM TensorCore GPUs, each with 80 GB of memory. For testing, we used a very large LLM (Llama 3.1 405B) and FP8 precision, which requires very fast accelerators and ample memory. With this LLM, which we augmented with in-house data, we quantified the number of supported users engaging in longer conversations to mirror what real-world users would experience. We used sample in-house data in conjunction with the LLM to address organizations that want the precision of a very large LLM using FP8 precision.”
In this testing, PT found that a single Dell PowerEdge XE9680 server with eight NVIDIA GPUs could support 68 simultaneous chatbot users engaging in lengthy questions and answers. The report goes on to note that “for a 60kW rack of six PowerEdge XE9680 servers, we estimate that such a solution could support 408 simultaneous users at a cost of approximately $8.3M over the next five years. If your organization needs the precision of a very large LLM using your own data, the Dell PowerEdge XE9680 server with NVIDIA H100 GPUs can provide ample resources to power your in-house chatbot.”
To learn more about PT testing on the Dell PowerEdge XE9680 server with NVIDIA H100 GPUs, read the full report at https://facts.pt/mFNE3Nh.
About Principled Technologies, Inc.
Principled Technologies, Inc. is the leading provider of technology marketing and learning & development services.
Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit www.principledtechnologies.com.
Sharon Horton
Principled Technologies, Inc.
press@principledtechnologies.com
Visit us on social media:
LinkedIn
Facebook
YouTube
X
Distribution channels: Business & Economy, IT Industry, Technology
Legal Disclaimer:
EIN Presswire provides this news content "as is" without warranty of any kind. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author above.
Submit your press release