Problem Statement¶

Business Context¶

The healthcare industry is rapidly evolving, with professionals facing increasing challenges in managing vast volumes of medical data while delivering accurate and timely diagnoses. The need for quick access to comprehensive, reliable, and up-to-date medical knowledge is critical for improving patient outcomes and ensuring informed decision-making in a fast-paced environment.

Healthcare professionals often encounter information overload, struggling to sift through extensive research and data to create accurate diagnoses and treatment plans. This challenge is amplified by the need for efficiency, particularly in emergencies, where time-sensitive decisions are vital. Furthermore, access to trusted, current medical information from renowned manuals and research papers is essential for maintaining high standards of care.

To address these challenges, healthcare centers can focus on integrating systems that streamline access to medical knowledge, provide tools to support quick decision-making, and enhance efficiency. Leveraging centralized knowledge platforms and ensuring healthcare providers have continuous access to reliable resources can significantly improve patient care and operational effectiveness.

Common Questions to Answer

1. Diagnostic Assistance: "What are the common symptoms and treatments for pulmonary embolism?"

2. Drug Information: "Can you provide the trade names of medications used for treating hypertension?"

3. Treatment Plans: "What are the first-line options and alternatives for managing rheumatoid arthritis?"

4. Specialty Knowledge: "What are the diagnostic steps for suspected endocrine disorders?"

5. Critical Care Protocols: "What is the protocol for managing sepsis in a critical care unit?"

Objective¶

As an AI specialist, your task is to develop a RAG-based AI solution using renowned medical manuals to address healthcare challenges. The objective is to understand issues like information overload, apply AI techniques to streamline decision-making, analyze its impact on diagnostics and patient outcomes, evaluate its potential to standardize care practices, and create a functional prototype demonstrating its feasibility and effectiveness.

Data Description¶

The Merck Manuals are medical references published by the American pharmaceutical company Merck & Co., that cover a wide range of medical topics, including disorders, tests, diagnoses, and drugs. The manuals have been published since 1899, when Merck & Co. was still a subsidiary of the German company Merck.

The manual is provided as a PDF with over 4,000 pages divided into 23 sections.

Installing and Importing Necessary Libraries and Dependencies¶

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.8 MB ? eta -:--:--
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 151.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 314.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 245.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 153.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 300.0 MB/s eta 0:00:00
  Building wheel for llama-cpp-python (pyproject.toml) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
langchain 0.3.0 requires numpy<2.0.0,>=1.26.0; python_version >= "3.12", but you have numpy 2.3.4 which is incompatible.
langchain-community 0.3.0 requires numpy<2.0.0,>=1.26.0; python_version >= "3.12", but you have numpy 2.3.4 which is incompatible.
chromadb 0.5.5 requires numpy<2.0.0,>=1.22.5, but you have numpy 2.3.4 which is incompatible.
cupy-cuda12x 13.3.0 requires numpy<2.3,>=1.22, but you have numpy 2.3.4 which is incompatible.
opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-exporter-otlp-proto-common==1.37.0, but you have opentelemetry-exporter-otlp-proto-common 1.38.0 which is incompatible.
opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-proto==1.37.0, but you have opentelemetry-proto 1.38.0 which is incompatible.
opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-sdk~=1.37.0, but you have opentelemetry-sdk 1.38.0 which is incompatible.
opencv-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.4 which is incompatible.
tensorflow 2.19.0 requires numpy<2.2.0,>=1.26.0, but you have numpy 2.3.4 which is incompatible.
google-adk 1.17.0 requires opentelemetry-api<=1.37.0,>=1.37.0, but you have opentelemetry-api 1.38.0 which is incompatible.
google-adk 1.17.0 requires opentelemetry-sdk<=1.37.0,>=1.37.0, but you have opentelemetry-sdk 1.38.0 which is incompatible.
opencv-python-headless 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.4 which is incompatible.
opencv-contrib-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.4 which is incompatible.
numba 0.60.0 requires numpy<2.1,>=1.22, but you have numpy 2.3.4 which is incompatible.

Note: The dependency errors or conflicts in the output can be ignored, as it will not impact scope and outcome of this project.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
opencv-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 1.26.4 which is incompatible.
thinc 8.3.6 requires numpy<3.0.0,>=2.0.0, but you have numpy 1.26.4 which is incompatible.
opencv-python-headless 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 1.26.4 which is incompatible.
jax 0.7.2 requires numpy>=2.0, but you have numpy 1.26.4 which is incompatible.
jaxlib 0.7.2 requires numpy>=2.0, but you have numpy 1.26.4 which is incompatible.
opencv-contrib-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 1.26.4 which is incompatible.
pytensor 2.35.1 requires numpy>=2.0, but you have numpy 1.26.4 which is incompatible.

Note:The error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.

LLM Selection for this project & Housekeeping¶

LLM Selection
I chose the following LLMs for this project:

  • The "TheBloke/Llama-2-13B-chat-GGUF" model, for building the RAG architecture and fine-tuning with different hyperparameters.
  • The "TheBloke/Mistral-7B-Instruct-v0.2-GGUF" model, which will serve as the "LLM-as-a-Judge" to evaluate responses from the Meta Llama RAG.

Housekeeping
LLMs calculate token size differently than humans do, because each LLM uses a unique tokenizer algorithm. For instance, the Llama-2-13B model typically considers one token to be approximately 4 characters long.

Downloading and instantiating the Meta's Llama-2-13B model¶

Configure Meta's Llama-2 with 13B parameters and 32 bit floating point, where 4 bit qunatization is applied to run the model locally with standard computing capacity.¶

Observation(s):

  • The path and base name for Meta Llama-2-13B-Chat has been configured.

Instantiating Meta's Llama-2 LLM with 13B parameters model using Llama class from Python wrapper.¶

Runtime parameters used are:

  • n_threads
  • n_batch
  • n_gpu_layer
  • context_window
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | 

Observation(s):

  • Meta Llama-2-13B-Chat LLM has been instantiated.

Define function to get response from the model based on user query (no RAG architecture yet)¶

Default parameters:¶

max_tokens = 512
temperature = 0
top_p = 0.95
top_k = 10

Answering 5 questions provided in the problem statement using default Llama-2-13B LLM (No RAG architecture, no fine-tuning, no prompt engineering applied here).¶

Question #1: What is the protocol for managing sepsis in a critical care unit?¶

Sepsis is a life-threatening condition that can arise from an infection, and it is a leading cause of death in hospitals. The management of sepsis in a critical care unit (CCU) requires a systematic approach that includes early recognition, prompt treatment, and close monitoring. Here is the protocol for managing sepsis in a CCU:

1. Early recognition:
	* Use a standardized screening tool, such as the Sepsis-3 criteria, to identify patients with suspected sepsis.
	* Monitor vital signs, particularly temperature, blood pressure, tachycardia, and tachypnea, every 4 hours or more frequently as needed.
	* Check for signs of organ dysfunction, such as altered mental status, confusion, or decreased urine output.
2. Rapid diagnostic testing:
	* Obtain blood cultures and other relevant tests (e.g., complete blood count, electrolytes, and coagulation studies) to confirm the diagnosis of sepsis.
	* Use a point-of-care lactate test to assess tissue perfusion and identify patients with severe sepsis or septic shock.
3. Resuscitation and supportive care:
	* Provide intravenous fluids, vasopressors, and oxygen therapy as needed to maintain mean arterial pressure (MAP) ≥65 mmHg and serum lactate levels <2 mmol/L.
	* Use a protocolized approach to titrate vasopressor therapy based on the patient's response to treatment.
	* Provide mechanical ventilation as needed to maintain adequate oxygenation and prevent respiratory failure.
4. Antimicrobial therapy:
	* Initiate broad-spectrum antibiotics within 1 hour of recognition of sepsis, based on the suspected etiology of infection.
	* Use a protocolized approach to guide antibiotic selection and dosing.
5. Source control:
	* Identify and manage the source of infection, such as removing a catheter or surgical drain, or performing surgical debridement if necessary.
6. Monitoring and escalation:
	* Closely monitor the

Observation(s):

  • The LLM has provided details on the protocol to manage sepsis in the critical care unit.
  • The response articulates the actions taken by doctors in the critical care unit.

Question #2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶

Llama.generate: prefix-match hit
Answer:

Appendicitis is a medical emergency that requires prompt treatment. The most common symptoms of appendicitis include:

1. Severe pain in the abdomen, usually starting near the belly button and then moving to the lower right side of the abdomen.
2. Nausea and vomiting.
3. Loss of appetite.
4. Fever.
5. Abdominal tenderness and guarding (muscle tension).
6. Abdominal swelling.

If you suspect that you or someone else has appendicitis, it is essential to seek medical attention immediately. Appendicitis cannot be cured via medicine; instead, surgery is required to remove the inflamed appendix. The most common surgical procedure for appendicitis is a laparoscopic appendectomy, where the surgeon makes several small incisions in the abdomen and uses a camera and specialized instruments to remove the inflamed appendix.

In some cases, an open appendectomy may be necessary, especially if the appendix has already ruptured or if there are other complications present. This procedure involves a larger incision in the abdomen to allow for better visualization of the appendix and surrounding tissues.

It is important to note that prompt treatment of appendicitis can help prevent complications such as the appendix rupturing, which can lead to infection of the abdominal cavity (peritonitis) and potentially life-threatening consequences. Therefore, if you experience any symptoms suggestive of appendicitis, do not hesitate to seek medical attention immediately.

Observation(s):

  • The LLM clearly articulated the common symptoms of appendicitis and the immediate first aid measures recommended upon their manifestation.
  • The response emphasized the necessity for patients experiencing symptoms to seek medical assistance as early as possible to mitigate the risk of further infection.
  • It was also noted that appendicitis cannot be definitively treated with medication and requires a surgical procedure for a permanent cure.

Question #3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶

Llama.generate: prefix-match hit
Sudden patchy hair loss, also known as alopecia areata, can be a distressing condition that affects millions of people worldwide. While there is no cure for this condition, there are several effective treatments and solutions that can help promote hair growth and reduce the size of bald spots. Here are some possible causes and treatment options:

Possible Causes:

1. Autoimmune disorder: Alopecia areata is believed to be an autoimmune disorder, where the immune system mistakenly attacks healthy hair follicles, leading to hair loss.
2. Hormonal imbalance: Hormonal changes, such as those that occur during pregnancy or menopause, can trigger alopecia areata.
3. Stress: Physical or emotional stress can contribute to the development of alopecia areata.
4. Genetics: A family history of alopecia areata increases the risk of developing the condition.
5. Skin conditions: Certain skin conditions, such as eczema or psoriasis, can increase the risk of developing alopecia areata.

Effective Treatments and Solutions:

1. Corticosteroid injections: Injecting corticosteroids into the affected area can help reduce inflammation and promote hair growth.
2. Topical treatments: Applying topical creams or ointments containing corticosteroids, minoxidil (Rogaine), or anthralin can help slow down hair loss and promote hair growth.
3. Phototherapy: Exposure to specific wavelengths of light, such as ultraviolet B (UVB) or narrowband UVB, can help reduce inflammation and promote hair growth.
4. Immunotherapy: This involves using medications to suppress the immune system and prevent it from attacking healthy hair follicles.
5. Wigs or hairpieces: For those with more extensive hair loss, wearing a wig or hairpiece can help restore a natural appearance and boost self-confidence.
6. Hair transplantation: In some cases, hair transplantation may be an option for restoring hair growth to the affected area.
7. Dietary changes: Eating a balanced diet rich in vit

Observation(s):

  • The model response articulates possible causes of sudden hair loss.
  • It also details effective treatments and solutions.

Question #4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶

Llama.generate: prefix-match hit
There are several treatment options available for individuals who have suffered a physical injury to brain tissue, depending on the severity and location of the injury. Here are some common treatments that may be recommended:

1. Medications: Depending on the type and severity of the injury, medications such as pain relievers, anti-inflammatory drugs, or anticonvulsants may be prescribed to manage symptoms such as pain, swelling, or seizures.
2. Rehabilitation therapy: Physical, occupational, and speech therapy may be necessary to help the individual regain lost function and improve their quality of life. These therapies can help improve cognitive, motor, and communication skills.
3. Surgery: In some cases, surgery may be necessary to relieve pressure on the brain or repair damaged tissue. This may involve procedures such as craniotomy (a procedure to remove a portion of the skull) or stereotactic surgery (a minimally invasive procedure using a frame-based system to guide the surgeon).
4. Stem cell therapy: Some research suggests that stem cells may have the potential to repair damaged brain tissue. However, this is still an experimental treatment and more research is needed to fully understand its effectiveness and safety.
5. Hyperbaric oxygen therapy: This involves breathing pure oxygen in a pressurized chamber, which can help improve blood flow and oxygenation to the brain. It may be recommended for individuals with certain types of brain injuries, such as those caused by carbon monoxide poisoning or decompression sickness.
6. Neurostimulation therapies: Techniques such as transcranial magnetic stimulation (TMS) and transcranial direct current stimulation (tDCS) may be used to improve brain function and reduce symptoms such as pain, fatigue, or depression.
7. Cognitive rehabilitation: This type of therapy can help individuals with brain injuries improve their cognitive skills, such as memory, attention, and problem-solving.
8. Psychological support: Brain injuries can have a significant impact on an individual's emotional well-being, and psychological support may be necessary to help them cope with the changes they are experiencing.
9. L

Observation(s):

  • The model response articulates some of the treatments recommended for a person who suffered brain injury.
  • The response ended with "9. L" without completing the sentence because it may have hit the token size specified using max_tokens.

Question #5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶

Llama.generate: prefix-match hit
A person who has fractured their leg while hiking may require immediate medical attention to ensure proper diagnosis, treatment, and management of the injury. The necessary precautions and treatment steps for such an individual include:

1. Immobilization: The affected limb should be immobilized using a splint or cast to prevent further movement and reduce pain.
2. Pain management: Pain medication may be prescribed to manage the discomfort and pain associated with the fracture.
3. Swelling reduction: Elevating the affected limb above heart level can help reduce swelling and improve blood flow.
4. Wound care: Any open wounds or cuts on the skin should be cleaned and covered with antimicrobial dressings to prevent infection.
5. Monitoring for complications: The individual should be monitored for signs of complications such as infection, nerve damage, or blood clots.
6. Rehabilitation: Once the initial pain and swelling have subsided, physical therapy may be necessary to help regain strength, flexibility, and range of motion in the affected limb.
7. Follow-up appointments: Regular follow-up appointments with a healthcare provider are essential to monitor progress, address any concerns or complications, and adjust treatment as needed.

Considerations for care and recovery may include:

1. Environmental factors: The individual should be kept in a clean, dry, and comfortable environment to facilitate healing.
2. Nutrition and hydration: A balanced diet and adequate hydration are essential for proper wound healing and recovery.
3. Rest and avoiding exacerbating activities: The individual should rest the affected limb and avoid any activities that may exacerbate the injury or delay healing.
4. Monitoring for signs of infection: Regular monitoring of the individual's temperature, pain levels, and wound drainage can help identify signs of infection early on.
5. Addressing any underlying medical conditions: Any underlying medical conditions such as diabetes or circulatory problems should be managed properly to prevent complications.
6. Psychological support: The emotional impact of a fracture, particularly during a hiking trip, should not be underestimated. Prov

Observation(s):

  • The model response articulates the precautions and treatments to be taken by a patient who has fractured their leg, and it also outlines the considerations for care and faster recovery.

Answering the questions using LLM with Prompt Engineering and parameter tuning.¶

Applying prompt engineering and LLM parameter tuning (at least 5 combinations) to answer 5 questions provided in the problem statement (No RAG architecture yet). Different combination of parameters/values is used for each question.

Question #1: What is the protocol for managing sepsis in a critical care unit?¶

Prompt Engineering and Parameter(s):

  1. Prompt Engineering: "You are an experienced medical professional. Provide only the factual steps directly, without introductory phrases."
  2. Parameter: temperature = 0.3 (highly deterministic with reduced randomness)
Llama.generate: prefix-match hit
1. Assess and monitor vital signs, including temperature, blood pressure, tissue perfusion, and oxygenation.
2. Administer broad-spectrum antibiotics and source control measures to manage or eliminate any infectious sources.
3. Provide vasopressor support if necessary to maintain mean arterial pressure (MAP) ≥65 mmHg.
4. Manage fluid resuscitation with crystalloid solutions, as needed.
5. Monitor and manage cardiac function, including electrocardiogram (ECG) and echocardiography.
6. Provide mechanical ventilation support if necessary.
7. Monitor and manage coagulopathy, including activated partial thromboplastin time (aPTT) and prothrombin time (PT).
8. Manage sepsis-induced organ dysfunction, including acute kidney injury (AKI), hepatic dysfunction, and respiratory failure.
9. Consider activated protein C (APC) therapy if clinically indicated.
10. Maintain a multidisciplinary approach to care, involving critical care physicians, nurses, and other healthcare professionals.

Observation(s):

  • By specifying the role of the LLM, the response is factual, crisp and concise, omitting paragraph headings or similar verbose formatting.
  • The response is easy to understand and written in plain language, avoiding excessive medical jargon.

Question #2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶

Prompt Engineering and Parameter(s):

  1. Prompt Engineering: "You are an experienced medical professional. Provide only the factual steps directly, without introductory phrases."
  2. Parameter: temperature = 0.8 (highly creative with increased randomness)
Llama.generate: prefix-match hit
Please note: The information provided is for general knowledge purposes only and should not be considered as professional medical advice, diagnosis or treatment. Always seek the advice of a qualified healthcare professional for any medical conditions.

Common symptoms of appendicitis include:

1. Severe pain in the lower right abdomen that begins suddenly and worsens over time.
2. Nausea and vomiting.
3. Loss of appetite.
4. Fever.
5. Abdominal tenderness and guarding (muscle tension) when the abdomen is touched.
6. Abdominal swelling.
7. Diarrhea or constipation.

Medications may be used to manage symptoms such as pain, fever, and nausea, but they do not cure appendicitis. The only effective treatment for appendicitis is surgical removal of the inflamed appendix, which is called an appendectomy.

The surgical procedure for treating appendicitis typically involves:

1. General anesthesia to ensure the patient is comfortable and pain-free during the procedure.
2. An incision in the lower right abdomen to access the inflamed appendix.
3. Removal of the inflamed appendix through the incision.
4. Closure of the incision with stitches or staples.
5. Monitoring for complications such as infection or bleeding.

Observation(s):

  • By specifying the role of the LLM, the response is factual, crisp and concise, omitting paragraph headings or similar verbose formatting.
  • The response is easy to understand and written in plain language, avoiding excessive medical jargon.
  • In addition, the surgical procedure outlines the high-level tasks performed before, during, and after the surgery, as requested in the user query.

Question #3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶

Prompt Engineering and Parameter(s):

  1. Prompt Engineering: "You are an experienced medical professional. Provide only the factual steps directly, without introductory phrases."
  2. Parameter: top_k = 5 (For next word prediction, consider only the top 5 matching words and ignore all other probable tokens.)
Llama.generate: prefix-match hit
Please provide a list of effective treatments or solutions for addressing sudden patchy hair loss, along with their potential causes.

Here are some effective treatments or solutions for addressing sudden patchy hair loss:

1. Medications: Minoxidil (Rogaine) and finasteride (Propecia) are two medications that have been approved by the FDA to treat hair loss. Minoxidil is a topical solution that stimulates hair growth and slows down hair loss, while finasteride is an oral medication that blocks the production of dihydrotestosterone (DHT), a hormone that contributes to hair loss.
2. Platelet-rich plasma (PRP) therapy: PRP therapy involves injecting platelet-rich plasma (PRP) into the scalp to stimulate hair growth. PRP is derived from the patient's own blood and contains growth factors that promote hair growth.
3. Low-level laser therapy (LLLT): LLLT uses low-level lasers or light-emitting diodes to stimulate hair growth. It is thought that the light energy increases blood flow to the scalp, promoting hair growth.
4. Hair transplantation: Hair transplantation involves moving healthy hair follicles from the back and sides of the head to the balding areas. This can help restore a natural-looking hairline and fill in balding areas.
5. Dietary changes: Making dietary changes such as increasing omega-3 fatty acid intake, reducing stress, and eating a balanced diet can help promote hair growth.
6. Reducing stress: Stress has been linked to hair loss, so reducing stress through relaxation techniques such as meditation or yoga can help promote hair growth.
7. Using a scalp massager: Massaging the scalp can increase blood flow and promote hair growth.
8. Applying a topical solution of saw palmetto: Saw palmetto has been shown to block the conversion of testosterone to dihydrotestosterone (DHT), which is a hormone that contributes to hair loss.
9. Using a hair growth supplement: Some hair growth supplements contain ingredients such as biotin, vitamin B, and ker

Observation(s):

  • By specifying the role of the LLM, the response is factual, crisp and concise, omitting paragraph headings or similar verbose formatting.
  • The response provides details on the treatment of sudden, patchy hair loss.
  • However, the user query had multiple parts, and the LLM failed to provide a response to the second part because the generation was truncated upon reaching the maximum token limit, which was consumed by the detailed response to the first part of the query.
  • Perhaps, more rigorous prompt engineering should have been applied to instruct the LLM to answer all parts within the given token limits when the question is multi-part.

Question #4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶

Prompt Engineering and Parameter(s):

  1. Prompt Engineering: "You are an experienced medical professional. Provide only the factual steps directly, without introductory phrases."
  2. Parameter: top_p = 0.9 (While predicting the next token, select the smallest set of most probable tokens that collectively make up 90% of the total probability mass.)
Llama.generate: prefix-match hit
Treatment options for a person with a brain injury may include:

1. Medications: To manage symptoms such as pain, anxiety, and depression.
2. Rehabilitation therapy: Physical, occupational, speech, and cognitive therapies to improve function and independence.
3. Surgery: To relieve pressure on the brain or repair damaged tissue.
4. Lifestyle changes: Modifying daily activities and environment to support recovery and prevent further injury.
5. Assistive technology: Using devices such as wheelchairs, walkers, and communication aids to enhance independence and improve quality of life.

Observation(s):

  • By specifying the role of the LLM, the response is factual, crisp and concise, omitting paragraph headings or similar verbose formatting.
  • The response provides details on the treatment of physical injury to brain tissues.
  • The response is complete in its entirety, indicating that it did not exceed the maximum token limit.

Question #5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶

Prompt Engineering and Parameter(s):

  1. Prompt Engineering: "You are an experienced medical professional. Provide only the factual steps directly, without introductory phrases."
  2. Parameter: max_tokens = 256 (Limit the generated response length to a maximum of 256 tokens.)
Llama.generate: prefix-match hit
1. Assess the severity of the injury: Check for any signs of neurovascular damage (numbness, tingling, or loss of sensation) or open wounds near the fracture site.
2. Immobilize the affected limb: Use a splint or cast to keep the leg stable and prevent further movement.
3. Manage pain: Administer medication as needed to manage pain and discomfort.
4

Observation(s):

  • By specifying the role of the LLM, the response is factual, crisp and concise, omitting paragraph headings or similar verbose formatting.
  • However, the response is incomplete even though the maximum token limit wasn't reached, which suggests the need for further investigation and tuning in terms of prompt engineering or parameterization.

Data Preparation for RAG¶

In this section, following functionalities are implemented

  • Load Medical diagnosis manual document (PDF)
  • Data overview, checking number of pages, and inspection by visualizing a couple of pages
  • Create semantic chunks using BPE tokenizer from OpenAI
  • Inspect a few chunks
  • Embedd the chunks
  • Build Medical Diagnosis Knowledge Base (KB)
  • Build Retriever mechanism

Load document¶

Load Medical diagnosis manual document (PDF)

Observation(s):

  • The "medical_diagnosis_manual.pdf" was loaded into med_diag_manual.

Data overview, checking number of pages, and inspection by visualizing a couple of pages¶

Visualize the first five pages of the PDF

Page Number : 1
mccombsaiml@icloud.com
B60CN7TIOG
ant for personal use by mccombsaiml@ic
shing the contents in part or full is liable 

Page Number : 2
mccombsaiml@icloud.com
B60CN7TIOG
This file is meant for personal use by mccombsaiml@icloud.com only.
Sharing or publishing the contents in part or full is liable for legal action.

Page Number : 3
Table of Contents
1
Front    ................................................................................................................................................................................................................
1
Cover    .......................................................................................................................................................................................................
2
Front Matter    ...........................................................................................................................................................................................
53
1 - Nutritional Disorders    ...............................................................................................................................................................
53
Chapter 1. Nutrition: General Considerations    .....................................................................................................................
59
Chapter 2. Undernutrition    .............................................................................................................................................................
69
Chapter 3. Nutritional Support    ...................................................................................................................................................
76
Chapter 4. Vitamin Deficiency, Dependency & Toxicity    ..................................................................................................
99
Chapter 5. Mineral Deficiency & Toxicity    ..............................................................................................................................
108
Chapter 6. Obesity & the Metabolic Syndrome    ...............................................................................................................
120
2 - Gastrointestinal Disorders    ..............................................................................................................................................
120
Chapter 7. Approach to the Patient With Upper GI Complaints    ...............................................................................
132
Chapter 8. Approach to the Patient With Lower GI Complaints    ...............................................................................
143
Chapter 9. Diagnostic & Therapeutic GI Procedures    ....................................................................................................
150
Chapter 10. GI Bleeding    ............................................................................................................................................................
158
Chapter 11. Acute Abdomen & Surgical Gastroenterology    .........................................................................................
172
Chapter 12. Esophageal & Swallowing Disorders    ..........................................................................................................
183
Chapter 13. Gastritis & Peptic Ulcer Disease    ..................................................................................................................
196
Chapter 14. Bezoars & Foreign Bodies    ..............................................................................................................................
199
Chapter 15. Pancreatitis    ............................................................................................................................................................
206
Chapter 16. Gastroenteritis    ......................................................................................................................................................
213
Chapter 17. Malabsorption Syndromes    ..............................................................................................................................
225
Chapter 18. Irritable Bowel Syndrome    ................................................................................................................................
229
Chapter 19. Inflammatory Bowel Disease    .........................................................................................................................
241
Chapter 20. Diverticular Disease    ...........................................................................................................................................
246
Chapter 21. Anorectal Disorders    ............................................................................................................................................
254
Chapter 22. Tumors of the GI Tract    ......................................................................................................................................
275
3 - Hepatic & Biliary Disorders    ............................................................................................................................................
275
Chapter 23. Approach to the Patient With Liver Disease    ...........................................................................................
294
Chapter 24. Testing for Hepatic & Biliary Disorders    ......................................................................................................
305
Chapter 25. Drugs & the Liver    ................................................................................................................................................
308
Chapter 26. Alcoholic Liver Disease    ....................................................................................................................................
314
Chapter 27. Fibrosis & Cirrhosis    ............................................................................................................................................
322
Chapter 28. Hepatitis    ..................................................................................................................................................................
333
Chapter 29. Vascular Disorders of the Liver    .....................................................................................................................
341
Chapter 30. Liver Masses & Granulomas    ..........................................................................................................................
348
Chapter 31. Gallbladder & Bile Duct Disorders    ...............................................................................................................
362
4 - Musculoskeletal & Connective Tissue Disorders    .........................................................................................
362
Chapter 32. Approach to the Patient With Joint Disease    ............................................................................................
373
Chapter 33. Autoimmune Rheumatic Disorders    ..............................................................................................................
391
Chapter 34. Vasculitis    .................................................................................................................................................................
416
Chapter 35. Joint Disorders    .....................................................................................................................................................
435
Chapter 36. Crystal-Induced Arthritides    ..............................................................................................................................
443
Chapter 37. Osteoporosis    .........................................................................................................................................................
448
Chapter 38. Paget's Disease of Bone    ..................................................................................................................................
451
Chapter 39. Osteonecrosis    .......................................................................................................................................................
455
Chapter 40. Infections of Joints & Bones    ...........................................................................................................................
463
Chapter 41. Bursa, Muscle & Tendon Disorders    .............................................................................................................
470
Chapter 42. Neck & Back Pain    ...............................................................................................................................................
481
Chapter 43. Hand Disorders    ....................................................................................................................................................
mccombsaiml@icloud.com
B60CN7TIOG
This file is meant for personal use by mccombsaiml@icloud.com only.
Sharing or publishing the contents in part or full is liable for legal action.

Page Number : 4
491
Chapter 44. Foot & Ankle Disorders    .....................................................................................................................................
502
Chapter 45. Tumors of Bones & Joints    ...............................................................................................................................
510
5 - Ear, Nose, Throat & Dental Disorders    ..................................................................................................................
510
Chapter 46. Approach to the Patient With Ear Problems    ...........................................................................................
523
Chapter 47. Hearing Loss    .........................................................................................................................................................
535
Chapter 48. Inner Ear Disorders    ............................................................................................................................................
542
Chapter 49. Middle Ear & Tympanic Membrane Disorders    ........................................................................................
550
Chapter 50. External Ear Disorders    .....................................................................................................................................
554
Chapter 51. Approach to the Patient With Nasal & Pharyngeal Symptoms    .......................................................
567
Chapter 52. Oral & Pharyngeal Disorders    .........................................................................................................................
578
Chapter 53. Nose & Paranasal Sinus Disorders    .............................................................................................................
584
Chapter 54. Laryngeal Disorders    ...........................................................................................................................................
590
Chapter 55. Tumors of the Head & Neck    ...........................................................................................................................
600
Chapter 56. Approach to Dental & Oral Symptoms    .......................................................................................................
619
Chapter 57. Common Dental Disorders    .............................................................................................................................
629
Chapter 58. Dental Emergencies    ..........................................................................................................................................
635
Chapter 59. Temporomandibular Disorders    ......................................................................................................................
641
6 - Eye Disorders    ............................................................................................................................................................................
641
Chapter 60. Approach to the Ophthalmologic Patient    ..................................................................................................
669
Chapter 61. Refractive Error    ...................................................................................................................................................
674
Chapter 62. Eyelid & Lacrimal Disorders    ...........................................................................................................................
680
Chapter 63. Conjunctival & Scleral Disorders    .................................................................................................................
690
Chapter 64. Corneal Disorders    ...............................................................................................................................................
703
Chapter 65. Glaucoma    ...............................................................................................................................................................
710
Chapter 66. Cataract    ...................................................................................................................................................................
713
Chapter 67. Uveitis    ......................................................................................................................................................................
719
Chapter 68. Retinal Disorders    .................................................................................................................................................
731
Chapter 69. Optic Nerve Disorders    ......................................................................................................................................
737
Chapter 70. Orbital Diseases    ..................................................................................................................................................
742
7 - Dermatologic Disorders    ....................................................................................................................................................
742
Chapter 71. Approach to the Dermatologic Patient    .......................................................................................................
755
Chapter 72. Principles of Topical Dermatologic Therapy    ............................................................................................
760
Chapter 73. Acne & Related Disorders    ...............................................................................................................................
766
Chapter 74. Bullous Diseases    .................................................................................................................................................
771
Chapter 75. Cornification Disorders    .....................................................................................................................................
775
Chapter 76. Dermatitis    ...............................................................................................................................................................
786
Chapter 77. Reactions to Sunlight    ........................................................................................................................................
791
Chapter 78. Psoriasis & Scaling Diseases    ........................................................................................................................
799
Chapter 79. Hypersensitivity & Inflammatory Disorders    .............................................................................................
808
Chapter 80. Sweating Disorders    ............................................................................................................................................
811
Chapter 81. Bacterial Skin Infections    ...................................................................................................................................
822
Chapter 82. Fungal Skin Infections    ......................................................................................................................................
831
Chapter 83. Parasitic Skin Infections    ...................................................................................................................................
836
Chapter 84. Viral Skin Diseases    ............................................................................................................................................
841
Chapter 85. Pigmentation Disorders    ....................................................................................................................................
846
Chapter 86. Hair Disorders    .......................................................................................................................................................
855
Chapter 87. Nail Disorders    .......................................................................................................................................................
861
Chapter 88. Pressure Ulcers    ...................................................................................................................................................
867
Chapter 89. Benign Tumors    .....................................................................................................................................................
874
Chapter 90. Cancers of the Skin    ............................................................................................................................................
882
8 - Endocrine & Metabolic Disorders    .............................................................................................................................
882
Chapter 91. Principles of Endocrinology    ............................................................................................................................
887
Chapter 92. Pituitary Disorders    ..............................................................................................................................................
901
Chapter 93. Thyroid Disorders    ................................................................................................................................................
mccombsaiml@icloud.com
B60CN7TIOG
This file is meant for personal use by mccombsaiml@icloud.com only.
Sharing or publishing the contents in part or full is liable for legal action.

Page Number : 5
921
Chapter 94. Adrenal Disorders    ................................................................................................................................................
936
Chapter 95. Polyglandular Deficiency Syndromes    ........................................................................................................
939
Chapter 96. Porphyrias    ..............................................................................................................................................................
949
Chapter 97. Fluid & Electrolyte Metabolism    .....................................................................................................................
987
Chapter 98. Acid-Base Regulation & Disorders    ..............................................................................................................
1001
Chapter 99. Diabetes Mellitus & Disorders of Carbohydrate Metabolism    ........................................................
1024
Chapter 100. Lipid Disorders    ................................................................................................................................................
1034
Chapter 101. Amyloidosis    ......................................................................................................................................................
1037
Chapter 102. Carcinoid Tumors    ..........................................................................................................................................
1040
Chapter 103. Multiple Endocrine Neoplasia Syndromes    .........................................................................................
1046
9 - Hematology & Oncology    ...............................................................................................................................................
1046
Chapter 104. Approach to the Patient With Anemia    ..................................................................................................
1050
Chapter 105. Anemias Caused by Deficient Erythropoiesis    ...................................................................................
1061
Chapter 106. Anemias Caused by Hemolysis    ...............................................................................................................
1078
Chapter 107. Neutropenia & Lymphocytopenia    ...........................................................................................................
1086
Chapter 108. Thrombocytopenia & Platelet Dysfunction    .........................................................................................
1097
Chapter 109. Hemostasis    ......................................................................................................................................................
1104
Chapter 110. Thrombotic Disorders    ...................................................................................................................................
1107
Chapter 111. Coagulation Disorders    ..................................................................................................................................
1113
Chapter 112. Bleeding Due to Abnormal Blood Vessels    ...........................................................................................
1116
Chapter 113. Spleen Disorders    ............................................................................................................................................
1120
Chapter 114. Eosinophilic Disorders    .................................................................................................................................
1126
Chapter 115. Histiocytic Syndromes    .................................................................................................................................
1131
Chapter 116. Myeloproliferative Disorders    .....................................................................................................................
1141
Chapter 117. Leukemias    .........................................................................................................................................................
1154
Chapter 118. Lymphomas    ......................................................................................................................................................
1164
Chapter 119. Plasma Cell Disorders    .................................................................................................................................
1172
Chapter 120. Iron Overload    ...................................................................................................................................................
1177
Chapter 121. Transfusion Medicine    ...................................................................................................................................
1186
Chapter 122. Overview of Cancer    ......................................................................................................................................
1198
Chapter 123. Tumor Immunology    .......................................................................................................................................
1204
Chapter 124. Principles of Cancer Therapy    ...................................................................................................................
1215
10 - Immunology; Allergic Disorders    ...........................................................................................................................
1215
Chapter 125. Biology of the Immune System    ...............................................................................................................
1227
Chapter 126. Immunodeficiency Disorders    ....................................................................................................................
1243
Chapter 127. Allergic & Other Hypersensitivity Disorders    .......................................................................................
1263
Chapter 128. Transplantation    ...............................................................................................................................................
1281
11 - Infectious Diseases    ........................................................................................................................................................
1281
Chapter 129. Biology of Infectious Disease    ...................................................................................................................
1300
Chapter 130. Laboratory Diagnosis of Infectious Disease    ......................................................................................
1306
Chapter 131. Immunization    ...................................................................................................................................................
1313
Chapter 132. Bacteria & Antibacterial Drugs    .................................................................................................................
1353
Chapter 133. Gram-Positive Cocci    ....................................................................................................................................
1366
Chapter 134. Gram-Positive Bacilli    ...................................................................................................................................
1376
Chapter 135. Gram-Negative Bacilli    .................................................................................................................................
1405
Chapter 136. Spirochetes    ......................................................................................................................................................
1413
Chapter 137. Neisseriaceae    .................................................................................................................................................
1419
Chapter 138. Chlamydia & Mycoplasmas    ......................................................................................................................
1421
Chapter 139. Rickettsiae & Related Organisms    ..........................................................................................................
1431
Chapter 140. Anaerobic Bacteria    ........................................................................................................................................
1450
Chapter 141. Mycobacteria    ...................................................................................................................................................
1470
Chapter 142. Fungi    ...................................................................................................................................................................
1493
Chapter 143. Approach to Parasitic Infections    .............................................................................................................
1496
Chapter 144. Nematodes (Roundworms)    .......................................................................................................................
mccombsaiml@icloud.com
B60CN7TIOG
This file is meant for personal use by mccombsaiml@icloud.com only.
Sharing or publishing the contents in part or full is liable for legal action.

Observation(s):

  • The first five pages of the document are displayed successfully.

Check number of pages in the PDF

4114

Observation(s):

  • The medical diagnosis manual comprises 4,114 pages.

Create semantic chunks using BPE tokenizer from OpenAI¶

Chunk using byte-pair (BPE) tokenizer with following parameters:

  • Encoder name: cl100K_base
  • chunk size = 512
  • chunk overlap = 50

Check the number of semantic chunks created by BPE

Number of chunks created: 8717

Observation(s):

  • The medical diagnosis manual was broken into chunks as expected, and the total number of semantic chunks created from 4,114 pages is 8,717.

Visualize the first semantic chunk

"The MERCK Manual\n1st Edition - 1899\n2nd Edition - 1901\n3rd Edition - 1905\n4th Edition - 1911\n5th Edition - 1923\n6th Edition - 1934\n7th Edition - 1940\n8th Edition - 1950\n9th Edition - 1956\n10th Edition - 1961\n11th Edition - 1966\n12th Edition - 1972\n13th Edition - 1977\n14th Edition - 1982\n15th Edition - 1987\n16th Edition - 1992\n17th Edition - 1999\n18th Edition - 2006\n19th Edition - 2011\nTHE MERCK INDEX\nFirst Edition, 1889\nTHE MERCK VETERINARY MANUAL\nFirst Edition, 1955\nTHE MERCK MANUAL OF GERIATRICS\nFirst Edition, 1990\nTHE MERCK MANUAL OF MEDICAL INFORMATION—HOME EDITION\nFirst Edition, 1997\nTHE MERCK MANUAL OF HEALTH & AGING\nFirst Edition, 2004\nTHE MERCK/MERIAL MANUAL FOR PET HEALTH\nFirst Edition, 2007\nTHE MERCK MANUAL OF PATIENT SYMPTOMS\nFirst Edition, 2008\nMerck books are published on a nonprofit basis as a service to the scientific community and the public.\nPreface\nAt the beginning of the 2nd decade of the 21st century, the amount of information available to health care\npractitioners is immense. Medical websites and journals send daily messages announcing results of the\nlatest studies. Within minutes, subspecialty data heretofore found only in university libraries can be\nunearthed, along with a vast array of information from academics, commercial organizations, advocacy\ngroups, the government, and seemingly anyone with a computer and an internet connection.\nWhat is the role of a general reference work such as The Merck Manual when seemingly the entire body\nof medical knowledge is at one's fingertips electronically? With such a vast body of knowledge available,\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nFront Matter\n4\nmccombsaiml@icloud.com\nB60CN7TIOG\nThis file is meant for personal use by mccombsaiml@icloud.com only."

Visualize the last semantic chunk

"Z\nZafirlukast 1879\nZalcitabine 1451\nin children 2854\nZaleplon 1709\nZanamivir 1407\nin influenza 1407, 1929\nZAP-70 (zeta-associated protein 70) deficiency 1092, 1108\nZavanelli maneuver 2680\nZellweger syndrome 2383, 3023\nZenker's diverticulum 125\nZidovudine 1451, 1453\nin children 2854\nZileuton 1881\nin asthma 1880\nZinc 49, 55, 3431-3432\nin common cold 1405\ndeficiency of 11, 49, 55\nin dermatophytoses 705\npoisoning with 3328, 3353\nrecommended dietary allowances for 50\nreference values for 3499\ntoxicity of 49, 55\ncopper deficiency and 49\nin Wilson's disease 52\nZinc oxide 2233\ngelatin formulation of 646, 672\nZinc pyrithione 647\nZinc shakes 55\nZipper injury 3239, 3240\nZiprasidone\nin agitation 1492\nin bipolar disorder 3059\npoisoning with 3347\nin schizophrenia 1566\nZoledronate 359, 361, 848\nZollinger-Ellison syndrome 95, 199, 200-201, 910\nmastocytosis vs 1125\nMenetrier's disease vs 132\npeptic ulcer disease vs 134\nZolmitriptan 1721\nZolpidem 1709, 3103\nZonisamide 1701\nZoonotic diseases, cutaneous 718\nZoophobia 1498\nZoster (see Herpes zoster virus infection)\nZygomycosis 1332\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nZ\n4104\nmccombsaiml@icloud.com\nB60CN7TIOG\nThis file is meant for personal use by mccombsaiml@icloud.com only.\nSharing or publishing the contents in part or full is liable for legal action."

Observation(s):

  • The semantic chunks were created successfully.

Instantiate "general text embedding large" model for embedding, embed a couple of chunks and verify the vector dimension.¶

Instantiate "general text embedding large" model for embedding

Observation(s):

  • The embedding model has been successfully instantiated.

Embed the first two semantic chunks

Print dimension of vectorized first chunk

Dimension of the embedding vector  1024

Observation(s):

  • The dimensions of the embedded vector database are 1,024 columns, which is as expected.

Compare dimensions of vectorized chunks 1 and 2

True

Build Knowledge Base (KB) vector¶

Define the path to create KB

Create KB by vectorizing semantic chunks

ERROR:chromadb.telemetry.product.posthog:Failed to send telemetry event ClientStartEvent: capture() takes 1 positional argument but 3 were given
ERROR:chromadb.telemetry.product.posthog:Failed to send telemetry event ClientCreateCollectionEvent: capture() takes 1 positional argument but 3 were given
HuggingFaceEmbeddings(client=SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
), model_name='thenlper/gte-large', cache_folder=None, model_kwargs={}, encode_kwargs={}, multi_process=False, show_progress=False)

Spot check if the embedding is successful

ERROR:chromadb.telemetry.product.posthog:Failed to send telemetry event CollectionQueryEvent: capture() takes 1 positional argument but 3 were given
[Document(metadata={'author': '', 'creationDate': 'D:20120615054440Z', 'creator': 'Atop CHM to PDF Converter', 'file_path': 'medical_diagnosis_manual.pdf', 'format': 'PDF 1.7', 'keywords': '', 'modDate': 'D:20251012195856Z', 'page': 2454, 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'source': 'medical_diagnosis_manual.pdf', 'subject': '', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'total_pages': 4114, 'trapped': ''}, page_content='to the organ involved.\nDiagnosis\nSepsis is suspected when a patient with a known infection develops systemic signs of inflammation or\norgan dysfunction. Similarly, a patient with otherwise unexplained signs of systemic inflammation should\nbe evaluated for infection by history, physical examination, and tests, including urinalysis and urine culture\n(particularly in patients who have indwelling catheters), serial blood cultures, and cultures of other\nsuspect body fluids. Blood levels of procalcitonin and C-reactive protein are elevated in severe sepsis\nand may facilitate diagnosis, but they are not specific. Ultimately, diagnosis is clinical.\nOther causes of shock (eg, hypovolemia, MI) should be sought via history, physical examination, ECG,\nand serum cardiac markers. Even without MI, hypoperfusion may result in ECG findings of ischemia\nincluding non-specific ST-T wave abnormalities, T-wave inversions, and supraventricular and ventricular\narrhythmias.\nCBC, ABGs, chest x-ray, serum electrolytes, lactate levels or sublingual PCO2, and liver function are\nmonitored. At the onset of septic shock, the WBC count may initially decrease to < 4000/μL, and PMNs\nmay be as low as 20%. However, this situation reverses within 1 to 4 h, and a significant increase in both\nthe total WBC count to > 15,000/μL and PMNs to > 80% (with predominantly juvenile forms) usually\noccurs. A sharp decrease in platelet count to ≤ 50,000/μL is often present early.\nHyperventilation with respiratory alkalosis (low PaCO2 and increased arterial pH) occurs early, in part as\ncompensation for lactic acidemia. Serum HCO3 is usually low, and serum and blood lactate levels\nincrease. As shock progresses, metabolic acidosis worsens, and blood pH decreases. Early respiratory\nfailure leads to hypoxemia with PaO2 < 70 mm Hg. Diffuse infiltrates may appear on the chest x-ray (see\nRespiratory Arrest on p. 2269). BUN and creatinine usually increase progressively as a result of renal\ninsufficiency. Bilirubin and transaminases may rise, although overt hepatic failure is uncommon.')]

Observation(s):

  • The medical diagnosis manual is converted into vectors based on semantic chunks.
  • As verified in the previous step, the dimension of the embedded vector database is 1,024 columns, which aligns with expectations.

Build Retriever Mechanism and test

Observation(s):

  • The retriever mechanism for the vector database, as part of RAG, has been established.

Answering the questions using LLM with RAG Architecture¶

Define necessary templates and a function to process user prompt and return a response back¶

Define template to capture, and combine system message and input prompt (user)¶

Setting the LLM's Role and Rules (aka System message)¶

Llama LLM's Role and Rules

Function to generate response based on user prompt and context provided¶

Query 1: What is the protocol for managing sepsis in a critical care unit?¶

/tmp/ipython-input-937327257.py:8: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 1.0. Use :meth:`~invoke` instead.
  context_for_llm = retriever.get_relevant_documents(query=input_prompt,k=3)
Llama.generate: prefix-match hit
Protocol for managing sepsis in a critical care unit:

1. Early recognition and aggressive fluid resuscitation with 0.9% saline until CVP reaches 8 mm Hg (10 cm H2O) or PAOP reaches 12 to 15 mm Hg.
2. Monitoring of systemic pressure, CVP, PAOP, pulse oximetry, ABGs, blood glucose, lactate, and electrolyte levels, renal function, and possibly sublingual PCO2.
3. Management of hypotension with dopamine if necessary, but avoid vasoconstriction caused by higher doses of dopamine and norepinephrine.
4. Empiric antibiotic therapy with a third-generation cephalosporin and gentamicin or tobramycin for septic shock of unknown cause.
5. Addition of vancomycin if resistant staphylococci or enterococci are suspected.
6. Surgical drainage of abscesses and excision of necrotic tissues.
7. Maintenance of normalization of blood glucose levels with continuous IV insulin infusion.

Observation(s):

  • The response from the LLM based on the RAG architecture appears to be more precise, as it was generated from the provided context.
  • The relevance, content, and tone of the LLM response seem to be tailored for doctors and other healthcare professionals, since the prompt provided clear instructions regarding the role of the LLM.

Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶

Llama.generate: prefix-match hit
Symptoms of Appendicitis:
     • Sudden severe pain on the lower right side of the abdomen
     • Nausea and vomiting
     • Fever
     • Loss of appetite
     • Abdominal tenderness and guarding (muscle tension)

Treatment:

If diagnosed early, appendicitis can be treated with antibiotics. However, if the appendix has burst or is perforated, surgery is necessary to prevent further complications. The surgical procedure for treating appendicitis is an appendectomy, which involves removing the inflamed appendix.

If the appendix has not burst or is not perforated, a laparoscopic appendectomy can be performed, where a small incision is made near the belly button and a laparoscope (thin tube with a camera) is inserted to visualize the appendix. The inflamed appendix is then removed through a smaller incision.

If the appendix has burst or is perforated, an open appendectomy may be necessary, where a larger incision is made in the abdomen to allow for better visualization of the affected area and drainage of any abscesses that may have formed.

Observation(s):

  • The symptoms appear to be the same as the answers generated by the LLM without RAG.
  • However, the treatment section appears to be more structured in the response above, using RAG. It clearly articulates the response based on three different scenarios: 1) early diagnosis, 2) appendix has not burst or perforated, and 3) appendix has burst or perforated.
  • The structured response is easy to read, understand, and follow.

Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶

Llama.generate: prefix-match hit
1. Topical corticosteroids.
2. Minoxidil (2% for women, 2% or 5% for men) prolongs the anagen growth phase and gradually enlarges miniaturized follicles into mature terminal hairs.
3. Finasteride inhibits the 5α-reductase enzyme, blocking conversion of testosterone to dihydrotestosterone, and is useful for male-pattern hair loss.

Observation(s):

  • Given that the top_k value was set to 3, the model worked as expected, providing the top 3 answers for effectively treating patchy hair loss.

Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶

Llama.generate: prefix-match hit
The following treatments may be recommended for a person who has sustained a physical injury to the brain tissue and is experiencing temporary or permanent impairment of brain function: 

1. Ensuring a reliable airway and maintaining adequate ventilation, oxygenation, and blood pressure. This can involve mechanical ventilation, oxygen therapy, and fluid replacement as needed.
2. Surgery to place monitors to track and treat intracranial pressure, decompress the brain if intracranial pressure is increased, or remove intracranial hematomas. 
3. Rehabilitation services such as physical, occupational, and speech therapy to regain lost function and prevent complications.
4. Skill-building activities and counseling to meet the patient's social and emotional needs. 
5. Medications to manage symptoms such as seizures, pain, or agitation. These may include anticonvulsants, analgesics, and sedatives. 
6. Family education and support to help the patient and their loved ones adjust to the changes caused by the injury. 
7. Prevention of secondary disabilities such as pressure ulcers, joint contractures, and pneumonia. 
8. Early intervention by rehabilitation specialists is indispensable for maximal functional recovery.

Observation(s):

  • The response received using RAG is more actionable from the healthcare professional's standpoint, whereas the response without RAG is more generic, using plain language without specific or actionable instructions for healthcare professionals.

Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶

Llama.generate: prefix-match hit
1. Immobilize the affected limb using a splint to prevent further injury or deformity.
2. Administer tetanus prophylaxis if necessary.
3. Provide pain management with analgesics as needed.
4. Monitor for signs of infection, such as increased pain, swelling, redness, or pus.
5. Ensure proper wound care and dressing to prevent infection and promote healing.
6. Consider early mobilization to minimize contractures and muscle atrophy.
7. Seek medical attention as soon as possible for proper evaluation and treatment.

Observation(s):

  • The LLM response using RAG is clearer, whereas the LLM response without RAG falls short of providing a complete set of details.

Fine-tuning of chunking, retriever and LLM parameters (at least 5). Answering questions using fine-tuned RAG¶

Parameters used to fine-tune - 8 in total

  • Chunking: chunking_Overlap=80 (from 50)
  • Retriever: k=5 (from 3)
  • Question #1: max_tokens=350 (from 512)
  • Question #2: top_p=0.85 (from 0.95)
  • Question #3: temperature=0.5 (from 0)
  • Question #4: top_k=2 (from 5)
  • Question #5: repeat_penalty=1.18 (from 1.15)

Create semantic chunks using BPE tokenizer from OpenAI¶

Chunk using byte-pair (BPE) tokenizer with following parameters:

  • Encoder name: cl100K_base
  • chunk size = 512
  • chunk overlap = 80 (from 50)

Observation(s):

  • The medical diagnosis manual was chunked with an 80-token overlap (i.e., context window), as opposed to the 50-token overlap defined earlier.

Check the number of semantic chunks created by BPE

Number of chunks created: 8945
"French—Editions d'Apres, Paris\nGerman—Elsevier, Ltd., Munich\nGreek—Medical & Scientific Publishing, Athens\nHungarian—Melania, Budapest\nItalian—Springer-Verlag Italia Srl (Medicom), Milan\nJapanese—Nikkei Business Publications, Tokyo\nKorean—Hanwoori Publishing Co., Seoul\nPolish—Elsevier, Ltd, Wroclaw\nPortuguese—Editora Roca Ltda, Sao Paolo\nRomanian—All Publishers, Bucharest\nRussian—Geotar-Media Publishing Group, Moscow\nSpanish—Elsevier Espana, S.A., Madrid\nTurkish—Yuce, Istanbul\nOther Merck Books\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nFront Matter\n3\nmccombsaiml@icloud.com\nB60CN7TIOG\nThis file is meant for personal use by mccombsaiml@icloud.com only.\nSharing or publishing the contents in part or full is liable for legal action."
"Z\nZafirlukast 1879\nZalcitabine 1451\nin children 2854\nZaleplon 1709\nZanamivir 1407\nin influenza 1407, 1929\nZAP-70 (zeta-associated protein 70) deficiency 1092, 1108\nZavanelli maneuver 2680\nZellweger syndrome 2383, 3023\nZenker's diverticulum 125\nZidovudine 1451, 1453\nin children 2854\nZileuton 1881\nin asthma 1880\nZinc 49, 55, 3431-3432\nin common cold 1405\ndeficiency of 11, 49, 55\nin dermatophytoses 705\npoisoning with 3328, 3353\nrecommended dietary allowances for 50\nreference values for 3499\ntoxicity of 49, 55\ncopper deficiency and 49\nin Wilson's disease 52\nZinc oxide 2233\ngelatin formulation of 646, 672\nZinc pyrithione 647\nZinc shakes 55\nZipper injury 3239, 3240\nZiprasidone\nin agitation 1492\nin bipolar disorder 3059\npoisoning with 3347\nin schizophrenia 1566\nZoledronate 359, 361, 848\nZollinger-Ellison syndrome 95, 199, 200-201, 910\nmastocytosis vs 1125\nMenetrier's disease vs 132\npeptic ulcer disease vs 134\nZolmitriptan 1721\nZolpidem 1709, 3103\nZonisamide 1701\nZoonotic diseases, cutaneous 718\nZoophobia 1498\nZoster (see Herpes zoster virus infection)\nZygomycosis 1332\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nZ\n4104\nmccombsaiml@icloud.com\nB60CN7TIOG\nThis file is meant for personal use by mccombsaiml@icloud.com only.\nSharing or publishing the contents in part or full is liable for legal action."

Observation(s):

  • The output of the basic exploratory data analysis looks good.

Instantiate "general text embedding large" model for embedding, embed a couple of chunks and verify the vector dimension.¶

Instantiate "general text embedding large" model for embedding

Embed the first two semantic chunks

Print dimension of vectorized first chunk

Dimension of the embedding vector  1024

Observation(s):

  • The embedding model was instantiated, and the vector has 1,024 dimensions (or columns).

Compare dimensions of vectorized chunks 1 and 2

True

Build Knowledge Base (KB) vector¶

Define the path to create KB

Create KB by vectorizing semantic chunks

ERROR:chromadb.telemetry.product.posthog:Failed to send telemetry event ClientStartEvent: capture() takes 1 positional argument but 3 were given
ERROR:chromadb.telemetry.product.posthog:Failed to send telemetry event ClientCreateCollectionEvent: capture() takes 1 positional argument but 3 were given
HuggingFaceEmbeddings(client=SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
), model_name='thenlper/gte-large', cache_folder=None, model_kwargs={}, encode_kwargs={}, multi_process=False, show_progress=False)

Spot check if the embedding is successful

[Document(metadata={'author': '', 'creationDate': 'D:20120615054440Z', 'creator': 'Atop CHM to PDF Converter', 'file_path': 'medical_diagnosis_manual.pdf', 'format': 'PDF 1.7', 'keywords': '', 'modDate': 'D:20251012195856Z', 'page': 2453, 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'source': 'medical_diagnosis_manual.pdf', 'subject': '', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'total_pages': 4114, 'trapped': ''}, page_content='Chapter 227. Sepsis and Septic Shock\nIntroduction\n(See also Ch. 226.)\nSepsis, severe sepsis, and septic shock are inflammatory states resulting from the systemic\nresponse to bacterial infection. In severe sepsis and septic shock, there is critical reduction in\ntissue perfusion. Common causes include gram-negative organisms, staphylococci, and\nmeningococci. Symptoms often begin with shaking chills and include fever, hypotension,\noliguria, and confusion. Acute failure of multiple organs, including the lungs, kidneys, and liver,\ncan occur. Treatment is aggressive fluid resuscitation, antibiotics, surgical excision of infected\nor necrotic tissues and drainage of pus, supportive care, and sometimes intensive control of\nblood glucose and administration of corticosteroids and activated protein C.\nA spectrum of severity exists (see\nTable 227-1).\nSepsis is infection accompanied by an acute inflammatory reaction with systemic manifestations\nassociated with release into the bloodstream of numerous endogenous mediators of inflammation. Acute\npancreatitis and major trauma, including burns, may manifest with signs of sepsis. The inflammatory\nreaction typically manifests with ≥ 2 of the following:\n• Temperature > 38°C or < 36°C\n• Heart rate > 90 beats/min\n• Respiratory rate > 20 breaths/min or PaCO2 < 32 mm Hg\n• WBC count > 12,000 cells/μL or < 4,000 cells/μL or > 10% immature forms\nHowever, these criteria are now viewed as suggestive but not sufficiently precise to be diagnostic.\nSevere sepsis is sepsis accompanied by signs of failure of at least one organ. Cardiovascular failure is\ntypically manifested by hypotension, respiratory failure by hypoxemia, renal failure by oliguria, and\nhematologic failure by coagulopathy.\nSeptic shock is severe sepsis with organ hypoperfusion and hypotension that are poorly responsive to\ninitial fluid resuscitation.\nEtiology\nMost cases of septic shock are caused by hospital-acquired gram-negative bacilli or gram-positive cocci\nand often occur in immunocompromised patients and patients with chronic and debilitating diseases.'),
 Document(metadata={'author': '', 'creationDate': 'D:20120615054440Z', 'creator': 'Atop CHM to PDF Converter', 'file_path': 'medical_diagnosis_manual.pdf', 'format': 'PDF 1.7', 'keywords': '', 'modDate': 'D:20251012195856Z', 'page': 2453, 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'source': 'medical_diagnosis_manual.pdf', 'subject': '', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'total_pages': 4114, 'trapped': ''}, page_content='Chapter 227. Sepsis and Septic Shock\nIntroduction\n(See also Ch. 226.)\nSepsis, severe sepsis, and septic shock are inflammatory states resulting from the systemic\nresponse to bacterial infection. In severe sepsis and septic shock, there is critical reduction in\ntissue perfusion. Common causes include gram-negative organisms, staphylococci, and\nmeningococci. Symptoms often begin with shaking chills and include fever, hypotension,\noliguria, and confusion. Acute failure of multiple organs, including the lungs, kidneys, and liver,\ncan occur. Treatment is aggressive fluid resuscitation, antibiotics, surgical excision of infected\nor necrotic tissues and drainage of pus, supportive care, and sometimes intensive control of\nblood glucose and administration of corticosteroids and activated protein C.\nA spectrum of severity exists (see\nTable 227-1).\nSepsis is infection accompanied by an acute inflammatory reaction with systemic manifestations\nassociated with release into the bloodstream of numerous endogenous mediators of inflammation. Acute\npancreatitis and major trauma, including burns, may manifest with signs of sepsis. The inflammatory\nreaction typically manifests with ≥ 2 of the following:\n• Temperature > 38°C or < 36°C\n• Heart rate > 90 beats/min\n• Respiratory rate > 20 breaths/min or PaCO2 < 32 mm Hg\n• WBC count > 12,000 cells/μL or < 4,000 cells/μL or > 10% immature forms\nHowever, these criteria are now viewed as suggestive but not sufficiently precise to be diagnostic.\nSevere sepsis is sepsis accompanied by signs of failure of at least one organ. Cardiovascular failure is\ntypically manifested by hypotension, respiratory failure by hypoxemia, renal failure by oliguria, and\nhematologic failure by coagulopathy.\nSeptic shock is severe sepsis with organ hypoperfusion and hypotension that are poorly responsive to\ninitial fluid resuscitation.\nEtiology\nMost cases of septic shock are caused by hospital-acquired gram-negative bacilli or gram-positive cocci\nand often occur in immunocompromised patients and patients with chronic and debilitating diseases.'),
 Document(metadata={'author': '', 'creationDate': 'D:20120615054440Z', 'creator': 'Atop CHM to PDF Converter', 'file_path': 'medical_diagnosis_manual.pdf', 'format': 'PDF 1.7', 'keywords': '', 'modDate': 'D:20251012195856Z', 'page': 2400, 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'source': 'medical_diagnosis_manual.pdf', 'subject': '', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'total_pages': 4114, 'trapped': ''}, page_content="16 - Critical Care Medicine\nChapter 222. Approach to the Critically Ill Patient\nIntroduction\nCritical care medicine specializes in caring for the most seriously ill patients. These patients are best\ntreated in an ICU staffed by experienced personnel. Some hospitals maintain separate units for special\npopulations (eg, cardiac, surgical, neurologic, pediatric, or neonatal patients). ICUs have a high\nnurse:patient ratio to provide the necessary high intensity of service, including treatment and monitoring\nof physiologic parameters.\nSupportive care for the ICU patient includes provision of adequate nutrition (see p. 21) and prevention of\ninfection, stress ulcers and gastritis (see p. 131), and pulmonary embolism (see p. 1920). Because 15 to\n25% of patients admitted to ICUs die there, physicians should know how to minimize suffering and help\ndying patients maintain dignity (see p. 3480).\nPatient Monitoring and Testing\nSome monitoring is manual (ie, by direct observation and physical examination) and intermittent, with the\nfrequency depending on the patient's illness. This monitoring usually includes measurement of vital signs\n(temperature, BP, pulse, and respiration rate), quantification of all fluid intake and output, and often daily\nweight. BP may be recorded by an automated sphygmomanometer; a transcutaneous sensor for pulse\noximetry is used as well.\nOther monitoring is ongoing and continuous, provided by complex devices that require special training\nand experience to operate. Most such devices generate an alarm if certain physiologic parameters are\nexceeded. Every ICU should strictly follow protocols for investigating alarms.\nBlood Tests\nAlthough frequent blood draws can destroy veins, cause pain, and lead to anemia, ICU patients typically\nhave routine daily blood tests to help detect problems early. Generally, patients need a daily set of\nelectrolytes and a CBC. Patients with arrhythmias should also have Mg, phosphate, and Ca levels\nmeasured. Patients receiving TPN need weekly liver enzymes and coagulation profiles. Other tests (eg,\nblood culture for fever, CBC after a bleeding episode) are done as needed.\nPoint-of-care testing uses miniaturized, highly automated devices to do certain blood tests at the patient's\nbedside or unit (particularly ICU, emergency department, and operating room). Commonly available tests")]

Observation(s):

  • The vector database (a.k.a. medical diagnosis knowledge base) has been built and is ready for use.

Build Retriever Mechanism by changing k value to 8 and test

Observation(s):

  • The retriever mechanism with a top_k value of 8 has been defined.

Query 1: What is the protocol for managing sepsis in a critical care unit?¶

Llama.generate: prefix-match hit
1. Monitor systemic pressure, CVP, PAOP, pulse oximetry, ABGs, blood glucose, lactate, and electrolyte levels frequently. 
2. Measure urine output with an indwelling catheter to assess renal perfusion. 
3. Provide fluid resuscitation with 0.9% saline until CVP reaches 8 mm Hg (10 cm H2O) or PAOP reaches 12-15 mm Hg. 
4. Monitor for oliguria and hypotension, and provide vigorous fluid resuscitation as needed. 
5. Use dopamine to increase mean BP to at least 60 mm Hg if the patient remains hypotensive after CVP or PAOP has been raised to target levels. 
6. Monitor for vasoconstriction caused by higher doses of dopamine and norepinephrine, and be aware of the risks of organ hypoperfusion and acidosis. 
7. Provide O2 therapy via mask or nasal prongs as needed. 
8. Consider tracheal intubation and mechanical ventilation for respiratory failure as needed.

Observation(s):

  • The response appears to be crisp and clear.
  • Highly suitable and relevant for healthcare professionals, with clear instructions on managing sepsis patients in a critical care unit.

Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶

Llama.generate: prefix-match hit
1. Severe pain in the abdomen that begins near the navel and then moves to the lower right area of the abdomen.
2. Nausea or vomiting
3. Abdominal tenderness
4. Fever
5. Loss of appetite

No, appendicitis cannot be cured via medicine. Surgery is required to treat it. The surgical procedure for treating appendicitis is an appendectomy, which involves the removal of the inflamed appendix. This can be done through laparoscopic or open surgery, depending on the severity of the condition and the patient's overall health.

Observation(s):

  • The response is crisp and clear.

Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶

Llama.generate: prefix-match hit
1. Alopecia Areata: This is a sudden patchy hair loss condition that can cause localized bald spots on the scalp. The exact cause of alopecia areata is not known, but it is believed to be an autoimmune disorder triggered by environmental factors.

2. Corticosteroid injections or topical corticosteroids: These medications can help suppress the immune system and promote hair growth. They may be prescribed for alopecia areata, lichen planopilaris, and other autoimmune conditions that cause patchy hair loss.

3. Topical minoxidil: This medication is applied to the scalp and can help stimulate hair growth by increasing blood flow to the hair follicles. It may be effective for alopecia areata, as well as other forms of patchy hair loss.

4. Oral antimalarials: These medications, such as piritanzer, may be prescribed for lichen planopilaris and other autoimmune conditions that cause patchy hair loss. They work by suppressing the immune system and reducing inflammation in the body.

5. Immunofluorescence studies: These tests can help diagnose autoimmune conditions such as lupus erythematosus, which may cause patchy hair loss.

6. Scalp biopsy: This test involves taking a small sample of skin from the scalp to examine under a microscope for signs of inflammation or scarring. It may be useful in diagnosing conditions such as lichen planopilaris or chronic cutaneous lupus erythematosus, which can cause patchy hair loss.

Observation(s):

  • The response clearly outlines effective treatments and solutions for addressing patchy hair loss.

Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶

Llama.generate: prefix-match hit
Recommended treatments for a person with a traumatic brain injury (TBI) depend on the severity and location of the injury. However, here are some common treatment recommendations:

1. Medication: To control seizures, reduce inflammation, and manage pain.
2. Rehabilitation therapy: Physical, occupational, speech, and cognitive therapies to help regain lost functions and improve daily living skills.
3. Cognitive training: To improve memory, attention, and other cognitive functions affected by the injury.
4. Psychological counseling: To address emotional and behavioral changes resulting from the injury.
5. Surgery: May be necessary to relieve pressure on the brain, repair skull fractures, or remove blood clots.
6. Rest and recovery: Adequate rest and sleep are crucial for the brain to recover from the injury.
7. Lifestyle modifications: Modifying daily activities, such as avoiding stressful situations, taking regular breaks, and pacing activities to prevent fatigue.
8. Assistive devices: Using assistive devices like canes, walkers, or wheelchairs to help with mobility and independence.
9. Family support: Encouraging family members to provide emotional support and assistance with daily tasks.
10. Follow-up care: Regular follow-up appointments with healthcare providers to monitor progress and make adjustments to the treatment plan as needed.

It is essential to note that each person's recovery from a TBI is unique, and the most effective treatment plan will depend on the individual's specific needs and circumstances. A healthcare professional should be consulted for personalized recommendations.

Observation(s):

  • The response clearly outlines effective treatments and solutions for managing brain tissue injury.

Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶

Llama.generate: prefix-match hit
1. Immobilization of the affected limb using a splint or cast to prevent further injury and promote healing.
2. Administration of analgesics and anti-inflammatory medications to manage pain and swelling.
3. Broad-spectrum antibiotics to prevent infection, especially if the fracture is open or dirty.
4. Tetanus prophylaxis to prevent tetanus infection.
5. Surgery may be necessary for unstable or complex fractures.
6. Elevation of the affected limb to reduce swelling.
7. Ice and compression wrapped around the affected limb to reduce pain and inflammation.
8. Non-weight-bearing exercises to maintain muscle strength and range of motion.
9. Crutches or a walker may be required for mobility assistance.

Observation(s):

  • The response clearly outlines precautions, and treatments for a person who fractured their leg.

LLM-as-a-judge strategy to evaluate response generated by Llama-2-13B using Mistral-7B model.¶

Let us now use the LLM-as-a-judge strategy to check the quality of the RAG system on two parameters - retrieval and generation. We illustrate this evaluation based on the answeres generated to the question from the previous section.

  • We are using Mistral-7B model for evaluation, so Llama-2-13B output is evaluated by Mistral-7B model.

Download and Load Mistral-7B model¶

Set model path

Download the model

Load Mistral-7B LLM

AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | 

Spot checking to validate Mistral-7B is up and running

'\n\nSepsis is a serious condition that can result from an infection anywhere in the body. It is characterized by the body’s response to an infection, which can include fever, chills, rapid heartbeat, and rapid breathing. In severe cases, sepsis can lead to tissue damage, organ failure, and even death.\n\nHere are some common symptoms of sepsis:\n\n1. Fever or feeling cold\n2. Rapid heart rate\n3. Rapid breathing\n4. Low blood pressure\n5. Confusion or disorientation\n6. Skin rashes or discolor'

Observation(s):

  • The LLM has been properly downloaded and instantiated.

Defining the Evaluation Prompts¶

Define system message for Llama-2-13B

Define system message for groundedness evaluation

Define system message for relevance evaluation

Define user message for Mistral-7B model

Define the Evaluation Function¶

Question #1: What is the protocol for managing sepsis in a critical care unit?¶

Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
 Steps to evaluate the answer:
1. Identify the information in the context related to managing sepsis in a critical care unit.
2. Compare each point in the AI generated answer with the corresponding information in the context.
3. Determine if the answer is derived solely from the context and not any external sources.

Explanation:
The AI generated answer adheres to the metric as it lists out the exact steps for managing sepsis in a critical care unit based on the information provided in the context. Each point in the answer can be traced back to specific details in the context, ensuring that the answer is derived only from the context.

Evaluation:
The metric is followed completely.

Rating:
Based on the evaluation criteria, the rating for this answer would be 5 (The metric is followed completely).

 Steps to evaluate the context as per the metric:
1. Identify the main aspects of the question, which are managing sepsis in a critical care unit.
2. Review the context to determine if it contains all and only the important aspects related to managing sepsis in a critical care unit.
3. Check if the context provides sufficient information for generating an answer that adheres to the metric.

The context adheres to the metric considering the question as the input because:
1. The context mentions all the essential steps for managing sepsis in a critical care unit, including early recognition and fluid resuscitation, monitoring of various physiologic parameters, management of fluid overload, use of vasopressors, O2 therapy, and normalization of blood glucose levels and corticosteroids.
2. The context provides sufficient detail for generating an answer that addresses the main aspects of the question based on the provided information.

The metric is followed mostly because:
1. The context covers all the important aspects related to managing sepsis in a critical care unit, including early recognition and fluid resuscitation, monitoring, management of fluid overload, use of vasopressors, O2 therapy, and normalization of blood glucose levels and corticosteroids.
2. The context provides sufficient detail for generating an answer that addresses the main aspects of the question based on the provided information.
3. The context is well-organized and easy to follow, making it straightforward to extract the relevant information needed to generate an answer.

Based on the evaluation criteria:
The metric is followed mostly (rating: 4) because the context covers all the important aspects related to managing sepsis in a critical care unit and provides sufficient detail for generating an answer that adheres to the metric.

Observation(s):

  • The LLM-as-a-Judge has rated the groundedness score of the Llama-2-13B LLM's response as 5, indicating that the response was generated based on the provided context. The Llama-2-13B LLM answered the question and adhered to the metric.
  • From a relevance metric standpoint, the response generated by the Llama-2-13B LLM mostly adhered to the metric, with a rating of 4.

Question #2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶

Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
 Steps to evaluate the answer:
1. Identify the information directly related to the question in the context.
2. Check if the AI generated answer only includes facts derived from the context.
3. Ensure that the answer does not include any additional or irrelevant information.

Explanation:
The AI generated answer adheres to the metric as it only includes facts derived directly from the context and does not add any additional or irrelevant information. The answer accurately states that appendicitis cannot be cured via medicine, and surgery is necessary to treat it, which is a fact stated in the context.

Evaluation:
The metric is followed completely.

Rating:
Based on the evaluation criteria, I would rate the answer as 5 (The metric is followed completely).

 To evaluate the context as per the metric of relevance, the following steps are needed:

1. Identify the main aspects of the question: common symptoms for appendicitis and whether it can be cured via medicine or if surgical intervention is required.
2. Determine if all and only the important aspects related to the question are contained in the context.
3. Check if the context provides sufficient information about the symptoms of appendicitis and whether medical treatment or surgery is necessary for its resolution.

The context adheres to the metric considering the question as the input because it contains comprehensive information on the diagnosis, treatment options, and symptoms of appendicitis. The context mentions that the treatment for appendicitis is surgical removal, which directly addresses the main aspects of the question regarding whether it can be cured via medicine or if surgery is required. Additionally, the context provides details about the symptoms of appendicitis, such as abdominal pain, anorexia, and tenderness, making it relevant to the first part of the question.

The metric is followed mostly in this context since all important aspects related to the question are addressed, including both the symptoms and treatment options for appendicitis.

Based on the evaluation criteria:
1 - The metric is not followed at all
2 - The metric is followed only to a limited extent
3 - The metric is followed to a good extent
4 - The metric is followed mostly
5 - The metric is followed completely

I would rate this context as a 4, since it follows the metric mostly.

Observation(s):

  • The LLM-as-a-Judge has rated the groundedness score of the Llama-2-13B LLM's response as 5, indicating that the response was generated based on the provided context. The Llama-2-13B LLM answered the question and adhered to the metric.
  • From a relevance metric standpoint, the response generated by the Llama-2-13B LLM mostly adhered to the metric, with a rating of 4.

Question #3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶

Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
 Steps to evaluate the answer:
1. Identify the main conditions discussed in the context that could cause sudden patchy hair loss.
2. Check if the AI generated answer mentions these conditions and their respective treatments.
3. Ensure that the answer derives information only from the context provided.

Explanation:
The AI generated answer adheres to the metric as it mentions various conditions causing sudden patchy hair loss, such as alopecia areata, lichen planopilaris, chronic cutaneous lupus erythematosus, telogen effluvium, androgenetic alopecia, fungal or bacterial infections, and scarring alopecia. The answer also provides the treatments for each condition based on the information provided in the context.

Rating:
Based on the evaluation criteria, I would rate the answer as a 5 because it follows the metric completely by deriving all the information from the context and providing accurate answers for various conditions causing sudden patchy hair loss along with their respective treatments.

 Steps to evaluate the context as per the metric:
1. Identify the main aspects of the question: Sudden patchy hair loss, causes, and effective treatments or solutions.
2. Determine if all important aspects are covered in the context: The context mentions various conditions that can cause sudden patchy hair loss (alopecia areata, lichen planopilaris, chronic cutaneous lupus erythematosus, telogen effluvium, androgenetic alopecia, fungal or bacterial infections, and scarring alopecia) and their respective treatments. It also discusses the causes of hair loss for some conditions (autoimmune disorders, hormonal imbalances, stress, and exposure to unclear environmental triggers).
3. Check if only important aspects are contained in the answer: The context does not include irrelevant information, making it relevant to the question.

The context adheres to the metric considering the question as the input.

Extent of following the metric:
4 - The metric is followed mostly

Rating based on the evaluation criteria:
Since the context covers all important aspects and only important aspects are contained in the answer, it can be rated as 5 (The metric is followed completely). However, since there's a slight possibility that some less common causes of sudden patchy hair loss might not be mentioned in the context, I would rate it as 4 (The metric is followed mostly) to account for this potential limitation.

Observation(s):

  • The LLM-as-a-Judge has rated the groundedness score of the Llama-2-13B LLM's response as 5, indicating that the response was generated based on the provided context. The Llama-2-13B LLM answered the question and adhered to the metric.
  • From a relevance metric standpoint, the response generated by the Llama-2-13B LLM mostly adhered to the metric, with a rating of 4.

Question #4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶

Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
 Steps to evaluate the answer:
1. Identify the information in the context related to treatments for brain injuries.
2. Compare the AI generated answer with the identified information from the context.
3. Determine if the answer is derived only from the information presented in the context.

Explanation:
The AI generated answer includes recommendations for treatments based on the context, such as rehabilitation planning, cognitive and emotional rehabilitation, monitoring of intracranial pressure, maintenance of adequate brain perfusion and oxygenation, and prevention of complications. These recommendations are directly derived from the information in the context. Therefore, the metric is followed completely.

Rating:
Based on the evaluation criteria, the answer receives a score of 5 for following the metric completely.

 Steps to evaluate the context as per the relevance metric:
1. Identify the main aspects of the question: The question asks about treatments for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function.
2. Determine if all important aspects are contained in the context: The context discusses various aspects related to traumatic brain injury (TBI), including diagnosis, initial treatment, and rehabilitation. It also mentions different types of injuries and their clinical manifestations. The context covers all the important aspects of the question as it discusses treatments for TBI patients in detail.

The context adheres to the relevance metric considering the question as the input because it addresses all the main aspects of the question, including the recommended treatments for a person with brain injury.

Extent to which the metric is followed: The metric is followed completely since the context covers all the important aspects of the question and provides detailed information about the recommended treatments for a person with brain injury.

Rating based on the evaluation criteria:
Since the context follows the relevance metric completely, it would receive a score of 5.

Observation(s):

  • The LLM-as-a-Judge has rated the groundedness score of the Llama-2-13B LLM's response as 5, indicating that the response was generated based on the provided context. The Llama-2-13B LLM answered the question and adhered to the metric.
  • From a relevance metric standpoint, the response generated by the Llama-2-13B LLM adhered to the metric, earning a rating of 5.

Question #5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶

Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
Llama.generate: prefix-match hit
 Steps to evaluate the answer:
1. Identify the information in the context that relates to the necessary precautions and treatment steps for a person who has fractured their leg.
2. Check if each element in the AI generated answer can be traced back to the context.
3. Evaluate the extent to which the answer adheres to the metric of being derived only from the information presented in the context.

Explanation:
The AI generated answer includes several steps that are necessary for the care and recovery of a person who has fractured their leg, as per the context provided. The context mentions immobilization using a splint or cast, tetanus prophylaxis, broad-spectrum antibiotics, sterile wound dressings, analgesics, crutches, ice and compression, stretching and strengthening exercises, monitoring for signs of infection, and follow-up appointments. Each of these steps is included in the AI generated answer. Therefore, the answer adheres to the metric as it is derived only from the information presented in the context.

Evaluation:
The metric is followed completely.

Rating:
Based on the evaluation criteria, the rating for this answer would be 5 (The metric is followed completely).

 Steps to evaluate the context as per the metric:
1. Identify the main aspects of the question: necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery.
2. Review the context to determine if it addresses all and only the important aspects of the question based on the information provided in the Merck Manual.
3. Check if the context mentions immobilization, which is a necessary precaution for a fractured leg. The context does mention immobilization through splinting or casting. (Relevant)
4. Verify if the context discusses treatment steps, such as tetanus prophylaxis and antibiotics for open fractures, analgesics for pain management, crutches for mobility aid, ice and compression, stretching and strengthening exercises, monitoring for signs of infection, and follow-up appointments. The context covers most of these aspects. (Mostly Relevant)
5. Consider if any irrelevant information is present in the context that does not address the main aspects of the question. In this case, there are no irrelevant details.

The metric is followed mostly as the context addresses most of the important aspects of the question related to precautions and treatment steps for a fractured leg. The answer generated by the AI system also follows the metric completely as it covers all the mentioned aspects in the context. Therefore, both the context and the answer receive a score of 5 (The metric is followed completely).

Observation(s):

  • The LLM-as-a-Judge has rated the groundedness score of the Llama-2-13B LLM's response as 5, indicating that the response was generated based on the provided context. The Llama-2-13B LLM answered the question and adhered to the metric.
  • From a relevance metric standpoint, the response generated by the Llama-2-13B LLM mostly adhered to the metric, earning a rating of 4.

Actionable Insights and Business Recommendations¶

Insights

  • Groundedness Metric

    • For all questions (i.e., 1, 2, 3, 4, and 5), the response from the Llama-2-13B-chat LLM achieved the highest groundedness score of 5 out of 5. This indicates that the model generated responses well-aligned with the provided context.
  • Relevance Metric

    • For all questions except #4 (i.e., 1, 2, 3, and 5), the relevance metric was mostly followed, with a score of 4 out of 5. This suggests that the LLM’s responses were generally relevant but could still be further optimized.
    • For question #4, the response fully adhered to the relevance metric, with a perfect score of 5 out of 5, indicating that the LLM’s response was fully relevant.
  • General Observations

    • Context Sensitivity: The more context supplied to the LLM (i.e., higher the top_k value), the greater the groundedness and relevance of the responses.
    • Vector Database Construction: The time taken to build the vector database was higher during the fine-tuning phase due to the increased token_overlap size (from 50 to 80 tokens).
    • Max Tokens Impact: Similarly, when the max_tokens value was higher (e.g., 1024), the LLM generated more detailed responses.

Recommendations

  • Review by the Business Team
    • The business team should review the prompts and responses for all five questions, providing feedback on the LLM's responses with respect to the user query, groundedness, and relevance metrics. This will ensure alignment with business requirements and refine the model’s performance.
  • System Prompt Optimization
    • It is likely that the LLM’s performance can be further improved by writing clearer and more precise system prompts. This could help the model fully address user queries, especially those that contain multiple parts or require more detailed responses.
  • Production and Monitoring
    • If the current metrics are deemed acceptable by the business team, the RAG can be productionized. It should then be monitored regularly for performance, with fine-tuning conducted as necessary to maintain or improve response quality over time.

Done!