Quantifying prevalence of use and biases in energy-related advice from Large Language models (QUBELL)

Large language models (LLMs), such as ChatGPT or Gemini, have become popular tools for information acquisition, including for consumer advice when making energy-related decisions. Yet how LLMs are used by the public in such contexts and what they can expect to obtain, depending on what information they provide in LLM prompts remains under-examined. These matter, since seemingly trivial prompt alterations, e.g. syntax, decimals or tone may unpredictably amend the output (see Figure 1). This risks making decisions based on expert-sounding yet incomplete or biased advice, compounded by equity issues: ability to appraise, IT skilfulness or (paid) access to premium LLMs. Project QUBELL intends to explore this gap in knowledge, ensure awareness-building and safeguarding, leading to safer use of LLMs for energy advice.

Figure 1: Difference in LLM’s (ChatGPT) outputs with trivial difference to the prompts – the context of heating appliance advice

Aim:

to improve understanding of prevalence and practices of consumers’ LLM use for energy decisions and to assess the quality of information obtained accordingly.

Objectives:

1. To perform rapid evidence review to establish state-of-knowledge and guide subsequent data collection and experiments.
2. To establish consumer segments concerning the use of and trust in LLMs for energy-related decisions using original survey data.
3. To assess how information quality and correctness of the LLM-generated advice on energy decisions is affected by information about consumer present in prompts.

Focused evidence review

Fast-paced emerging evidence review concerning use of bots, LLMs and similar tools for information acquisition, including possible biases and their impacts in energy-related contexts.

Online survey

Survey of a sample of individuals (online panel, UK) focused on practices of LLM use for different purposes, including energy advice, trust and impact of such advice on decisions and knowledgeableness on energy-related matters.

LLM experiments

Systematic evaluation of at least three cloud-based LLMs using experiments with scenarios of consumer seeking advice on:

1. Energy bill reduction;
2. Energy tariff selection;
3. Participation in demand-side response.

As part of the project outputs we intend to deliver the following items by mid-2026:

• a peer reviewed journal paper, presenting results of the survey analysis as well as the LLM bias analysis;
• EDRC Policy brief/White paper: a policy-oriented of the results (with journal paper details as technical appendices where necessary), to be published via EDRC website and as part of the dissemination activities below;
• survey data, formatted and documented, deposited in a repository for analysis by other researchers;
• dissemination activities: presentation at EDRC all-hands; popularisation via social media channels of EDRC and ICL.

We hope that project results can help in four areas of impact.

Specifically, the findings can assist policymakers and regulators in understanding how the use of LLM may affect people’s energy decisions, including identifying any potential misinformation risks, but also opportunities in using LLMs for improved awareness and tailored advice.

Moreover, the project will facilitate further research opportunities at the intersection of the rapidly expanding fields of LLM and AI, including providing data assets for further analysis.

Lastly, the project can also contribute towards positive environmental impact, looking at LLMs as a factor potentially helping people achieve reduced energy consumption.