On Sunday, CBS 60 Minutes presented a segment about Khanmigo, an AI tutor that is powered by LLM technology. It has shown some very impressive results, and the teachers who use it in their classes have found it very helpful. It doesn't replace teachers. It helps them by offloading routine testing and tutoring,
https://www.cbsnews.com/video/khanmigo-ai-tutor-60-minutes-video-2024-12-08/
As I have said many times. there are serious limitations to the LLM technology, which requires evaluation to avoid serious errors and hallucinogenic disasters. Question; How can Khanmigo and related systems avoid those disasters?
I do not know the details of the Khanmigo implementation. But from the examples they showed, I suspect that they avoid mistakes by (1) Starting with a large text that was written, tested, and verified by humans (possibly with some computer aid); (2) For each topic, the system does Q/A primarily by translation; (3) And the LLM technology was first developed for translation and Q/A; (4) if the source text is tested and verified, a Q/A system that is based on that text can usually be very good and dependable.
But the CBS program did show an example where the system made some mistakes.
Summary: This example shows great potential for the LLM technology. But it also shows the need for evaluation by the traditional AI symbolic methods. Those methods have been tried and tested for over 50 years, and they are just as important today as they ever were.
As a reminder: LLMs can be used with a large volume of sources to find information and to generate hypotheses. But if the source is very large and unverified for accuracy, it can and does find and generate erroneous or even dangerously false information. That is why traditional AI methods are essential for evaluating what they find in a large volume of source data.
Danger; The larger the sources, the more likely that the LLMs will find bad data. Without evaluation, bigger is definitely not better. I am skeptical about attempts to create super large volumes of LLM data. Those systems consume enormous amounts of electricity with a diminishing return on investment.
There is already a backlash by employees of Google and Elon M.
John
--------------------------------------------------
CALL FOR PAPERS
HHAI 2025 - Hybrid Human-Artificial Intelligence
https://hhai-conference.org/2025/
June 9–13, 2025, Pisa, Italy
--------------------------------------------------
Hybrid Human-Artificial Intelligence (HHAI) is an international conference series that focuses on the study of Artificial Intelligence systems that cooperate synergistically, proactively and purposefully with humans, amplifying instead of replacing human intelligence. HHAI aims for AI systems that work together with humans, emphasizing the need for adaptive, collaborative, responsible, interactive and human-centered intelligent systems. HHAI systems leverage human strengths and compensate for human weaknesses, while taking into account social, ethical and legal considerations.
HHAI 2025 will be held on June 9–13, 2025, in Pisa, Italy, and is the fourth conference in the series. The HHAI field is driven by developments in AI, but it also requires fundamentally new approaches and solutions. Thus, we encourage collaborations across research domains such as AI, HCI, cognitive and social sciences, philosophy and ethics, complex systems, and others. In this fourth international conference, we invite scholars from these fields to submit their best original – new as well as in progress – works, and visionary ideas on Hybrid Human-Artificial Intelligence.
**Join the HHAI community and keep up with the news:**
Website: https://hhai-conference.org/2025/
Linkedin: https://www.linkedin.com/company/hhai-conference/
X: https://x.com/hhai_conference
Mastodon: https://sigmoid.social/@hhai
IMPORTANT DATES
Abstract submission: January 17th, 2025
Paper submission: January 24th, 2025
Acceptance notification: March 16th, 2025
Camera-ready version: April 13th, 2025
Conference: June 9-13, 2025
LOCATION
HHAI 2025 will be an in-person, single-track conference organized in Pisa, Italy. Workshops and tutorials (9-10 June) will be held at the University of Pisa and Scuola Normale Superiore in Pisa, Italy. The main conference (11-13 June) will be held at CNR.
TOPICS
We invite research on different challenges in Hybrid Human-Artificial Intelligence. The following list of topics is illustrative, not exhaustive:
- Human-AI interaction, interpretation and collaboration
- Adaptive human-AI co-learning and co-creation
- Learning, reasoning and planning with humans and machines in the loop
- User modeling and personalisation
- Integration of learning and reasoning
- Transparent, explainable, and accountable AI
- Fair, ethical, responsible, and trustworthy AI
- Societal awareness of AI
- Multimodal machine perception of real-world settings
- Social signal processing
- Representations learning for Communicative or Collaborative AI
- Symbolic representations for human-centric AI
- Human-AI Coevolution
- Foundation models and humans
- Human cognition-aware AI
- Decentralized human-AI systems
- Reliability and robustness in human-AI systems
- Applications of hybrid human-AI intelligence
We welcome contributions about all types of technology, from robots and conversational agents to multi-agent systems and machine learning models.
PAPER TYPES
In this conference, we wish to stimulate the exchange of novel ideas and interdisciplinary perspectives. To do this, we will accept three different types of papers:
- Full papers present original, impactful work (12 pages excluding references)
- Blue sky papers present visionary ideas to stimulate the research community (8 pages excluding references)
- Working papers present work in progress (8 pages excluding references)
Accepted full papers and Blue sky papers will be published in the Proceedings of the Fourth International Conference on Hybrid Human-Machine Intelligence, in the Frontiers of AI series by IOS Press. Working papers can be included in these proceedings, unless the authors request the paper to remain unpublished.
REVIEWING PROCESS & SUBMISSION GUIDELINES
Submissions of full, blue-sky, and working papers should be original work without substantial overlap with pre-published papers. All submissions should adhere to IOS formatting guidelines. Papers should be written in English and detailed submission instructions can also be found here.
**Important**
HHAI 2025 will follow a double-blind reviewing process. Thus, submissions must exclude all information that might disclose the authors’ names or affiliations.
All studies involving human participants should have received human-research ethics consent from the relevant institutions and mention this in the paper.
Work should be submitted in PDF format via Easychair (link to be announced soon).
On acceptance, at least one author should attend the conference. A significant contribution is expected from all authors.
PROGRAM CHAIRS
Chiara Boldrini (IIT-CNR, IT)
Luca Pappalardo (ISTI-CNR, IT)
Andrea Passerini (University of Trento, IT)
Shenghui Wang (University of Twente, NL)
CONFERENCE CHAIRS
Michela Milano (University of Bologna, IT)
Dino Pedreschi (University of Pisa, IT)
Stuart Russell (University of California Berkeley, US)
Ilaria Tiddi (Vrije Universiteit Amsterdam, NL)
CONTACT INFORMATION
For questions, you can reach the program chairs at: program(a)hhai-conference.org
The QwQ system combines LLM technology with traditional AI methods to do the evaluation. This is a hybrid technique that our Permion.ai system uses.
I don't know anything more that I read in the in the following text and the link to a more detailed article. But I believe that hybrid methods are essential for developing reliable and trustworthy AI systems.
John
----------------------
QwQ-32B is an experimental AI model designed to approach problem-solving with deep introspection, emphasizing questioning and reflection before reaching conclusions. Despite its limitations, including language-switching issues and recursive reasoning loops, QwQ demonstrates impressive capabilities in areas like mathematics and coding. For AI practitioners, QwQ represents an attempt to embed a philosophical dimension into reasoning processes, striving for deeper and more robust outcomes—important for teams aiming to build AI that is both effective and adaptable.
QwQ: Reflect Deeply on the Boundaries of the Unknown
https://qwenlm.github.io/blog/qwq-32b-preview
What does it mean to think, to question, to understand? These are the deep waters that QwQ (Qwen with Questions) wades into. Like an eternal student of wisdom, it approaches every problem - be it mathematics, code, or knowledge of our world - with genuine wonder and doubt. QwQ embodies that ancient philosophical spirit: it knows that it knows nothing, and that’s precisely what drives its curiosity. Before settling on any answer, it turns inward, questioning its own assumptions, exploring different paths of thought, always seeking deeper truth. Yet, like all seekers of wisdom, QwQ has its limitations. This version is but an early step on a longer journey - a student still learning to walk the path of reasoning. Its thoughts sometimes wander, its answers aren’t always complete, and its wisdom is still growing. But isn’t that the beauty of true learning? To be both capable and humble, knowledgeable yet always questioning? We invite you to explore alongside QwQ, embracing both its insights and its imperfections as part of the endless quest for understanding.
Limitations
QwQ-32B-Preview is an experimental research model developed by the Qwen Team, focused on advancing AI reasoning capabilities. As a preview release, it demonstrates promising analytical abilities while having several important limitations:
- Language Mixing and Code-Switching: The model may mix languages or switch between them unexpectedly, affecting response clarity.
- Recursive Reasoning Loops: The model may enter circular reasoning patterns, leading to lengthy responses without a conclusive answer.
- Safety and Ethical Considerations: The model requires enhanced safety measures to ensure reliable and secure performance, and users should exercise caution when deploying it.
- Performance and Benchmark Limitations: The model excels in math and coding but has room for improvement in other areas, such as common sense reasoning and nuanced language understanding.
Performance
Through deep exploration and countless trials, we discovered something profound: when given time to ponder, to question, and to reflect, the model’s understanding of mathematics and programming blossoms like a flower opening to the sun. Just as a student grows wiser by carefully examining their work and learning from mistakes, our model achieves deeper insight through patient, thoughtful analysis. This process of careful reflection and self-questioning leads to remarkable breakthroughs in solving complex problems. Our journey of discovery revealed the model’s exceptional ability to tackle some of the most challenging problems in mathematics and programming, including:
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark, a challenging benchmark for evaluating scientific problem-solving abilities through grade school level questions.
- AIME: American Invitation Mathematics Evaluation, which tests mathematical problem solving with arithmetic, algebra, counting, geometry, number theory, and probability and other secondary school math topics.
- MATH-500: The 500 test cases of the MATH benchmark, a comprehensive dataset testing mathematical problem-solving.
- LiveCodeBench: A challenging benchmark for evaluating code generation and problem solving abilities in real-world programming scenarios.
Mike and Igor,
I'm glad that you mentioned design patterns. But instead of an ontology of design patterns, I would say that design patterns should be the BASIS for ontology. In fact, I would point out that visualization and conceptualization are the basis for mathematics. The formal notation always comes at the end, never at the beginning.
The term 'design pattern' is for the kinds of visualizations that mathematicians and logicians start with. The formal notations are essential, but creative mathematicians always start with visual patterns long before they get down to the formal details. In fact, mathematicians usually have the problem completely solved in diagrams long before they work out the formal notations.
The formalism is essential to verify that the visualization is correct and to specify every last detail. And the task of writing out the formal details can often point out issues that were missing or mistaken in the visualization. Visualization is the essence of mathematics. Formalization is the housekeeping that is important for keeping everything neat and tidy.
More generally, I would emphasize the trio of Conceptualization, Analysis, and Formalization. We need all three. For comments by Paul Halmos (a former president of the American Mathematical Society) and Albert Einstein, see the following slide 27. For more discussion and references, see https://jfsowa.com/talks/eswc.pdf .
----------------------------------------
From: "Mike Bennett" <mbennett(a)hypercube.co.uk>
A design pattern ontology would be a very different thing to an ontology
design pattern, but both are things of value.
At the Object Management Group (OMG) we maintain a suite of standards
based on many of these design patterns, i.e. UML and the underpinnings
in MOF.
We are also exploring whether or how to move some of these modeling
languages from being MOF-based to using the new more semantically rich
framework that has been developed for the SysML V2 standards (which has
a kernel language called KerML). These are still in the Finalization
process.
This is where the distinction between model semantics and the semantics
of the target problem domain subject matter become an important
consideration. For example MOF was all about model element semantics.
Ideally some of these directions will move things towards something with
clear formal semantics both for model semantics and how subject matter
semantics is treated. Whether that's in KerML or a more conventional
ontology standard such as FOL or DL, or a syntax such as RDF/OWL,
remains to be seen.
If anyone did happen to be doing a formal ontology of these software
design patterns, this would be very helpful to know.
Meanwhile you should probably also check out the OMG's ESSENCE standard
(spearheaded by Ivar Jacobson) for the kind of model concepts needed to
model a design methodology.
Mike
On 11/25/2024 7:41 PM, 'Igor Toujilov' via ontolog-forum wrote:
> Hi All,
> I am studying a famous book [1] written by the Gang of Four. I am
> surprised that despite it being written 30 years ago, I did not find a
> design pattern ontology. There is plenty of material on ontology
> design patterns on the Internet, but nothing about a design pattern
> ontology which I miss and want to create if it does not exist yet.
> Please advise if I overlooked something.
>
> Regards,
> Igor
>
> [1] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design
> patterns : elements of reusable object-oriented software. 1994.
Differential Logic • 1
• https://inquiryintoinquiry.com/2024/10/30/differential-logic-1-a/
Introduction —
Differential logic is the component of logic whose object is
the description of variation — focusing on the aspects of change,
difference, distribution, and diversity — in universes of discourse
subject to logical description. A definition that broad naturally
incorporates any study of variation by way of mathematical models,
but differential logic is especially charged with the qualitative
aspects of variation pervading or preceding quantitative models.
To the extent a logical inquiry makes use of a formal system, its
differential component governs the use of a “differential logical
calculus”, that is, a formal system with the expressive capacity
to describe change and diversity in logical universes of discourse.
Simple examples of differential logical calculi are furnished by
“differential propositional calculi”. A differential propositional
calculus is a propositional calculus extended by a set of terms for
describing aspects of change and difference, for example, processes
taking place in a universe of discourse or transformations mapping
a source universe to a target universe. Such a calculus augments
ordinary propositional calculus in the same way the differential
calculus of Leibniz and Newton augments the analytic geometry of
Descartes.
Resources —
Survey of Differential Logic
• https://inquiryintoinquiry.com/2024/02/25/survey-of-differential-logic-7/
Regards,
Jon
cc: https://www.academia.edu/community/lJX2qa
cc: https://www.researchgate.net/post/Differential_Logic_The_Logic_of_Change_an…
Pascal,
I just read your paper (cited below). I agree that LLM technology is good for finding important and valuable information. But as you know, there are serious issues about evaluating that information to avoid irrelevant, erroneous, or even hallucinogenic data. I didn't see much attention devoted to evaluation and testing.
As I often mention, our old VivoMind company was doing large volumes of high-speed knowledge extraction, analysis, evaluation, and processing over 20 years ago. For a description of that system with some examples of large applications, see https://jfsowa.com/tallks/cogmem.pdf . The systems described there are just a small sample of the applications, since our customers do not want their data or methods publicized.
I also noticed that you are using OWL for ontology. We use a high-speed version of Prolog, which is much richer, more powerful, and faster than OWL, which implements a tiny subset of the logic that Tim Berners-Lee had proposed for the Semantic Web.
Some of our customers were among the sponsors of the IKRIS project, funded from 2004 to 2006, to support a much larger and more powerful version of what Tim BL had proposed. For an overview of IKRIS with links to some of the original publications, see https://jfsowa.com/ikl .
The IKL technology does not replace LLM, but it is valuable for evaluating the results generated by LLM, detecting errors and avoiding irrelevant, erroneous, or even hallucinogenic data. When processing high volumes of data at high speed, human checking is not possible. High quality computer checking is necessary to eliminate 99% or more of the bad or even dangerous data.
Human checking would only be required for the tiny percentage of data for which the computational methods are uncertain. For a more recent talk, see https://jfsowa.com/talks/eswc.pdf .
John
----------------------------------------
From: "Pascal Hitzler' via ontolog-forum" <ontolog-forum(a)googlegroups.com>
Given the currently ongoing ISWC2024 conference and all the discussions around this neurosymbolic topic: Link to our (with Cogan Shimizu) position paper on this: https://kastle-lab.github.io/assets/publications/2024-LLMs4KGOE.pdf
The developments are really exciting!
Pascal.
Interesting question.
John
____________________
There may be a cosmic speed limit on how fast anything can grow
Alan Turing's theories about computation seem to have a startling consequence, placing hard limits on how fast or slow any physical process in the universe can grow
https://www.newscientist.com/article/2454024-there-may-be-a-cosmic-speed-li…
A newly proposed cosmic speed limit may constrain how fast anything in the universe can grow. Its existence follows from Alan Turing’s pioneering work on theoretical computer science, which opens the intriguing possibility that the structure of the universe is fundamentally linked to the nature of computation.
Cosmic limits aren’t a new idea. While studying the relationship between space and time, Albert Einstein showed that nothing in the universe can exceed the speed of light, as part of his special theory of relativity. Now, Toby Ord at the University of Oxford is proposing a new physical limit based on computation.
“I had the seed of this idea more than 20 years ago,” he says. “It would apply to any quantity you can directly measure, including mass, charge, energy, etc., and even more subtle things like the time intervals between a sequence of events.” . . .
Information = Comprehension × Extension • Comment 1
• https://inquiryintoinquiry.com/2024/10/11/information-comprehension-x-exten…
Re: Information = Comprehension × Extension • Selection 1
• https://inquiryintoinquiry.com/2024/10/05/information-comprehension-x-exten…
All,
Selection 1 ends with Peirce drawing the following conclusion about the
links between information, comprehension, inference, and symbolization.
❝Thus information measures the superfluous comprehension.
And, hence, whenever we make a symbol to express any thing
or any attribute we cannot make it so empty that it shall
have no superfluous comprehension.
❝I am going, next, to show that inference is symbolization
and that the puzzle of the validity of scientific inference
lies merely in this superfluous comprehension and is therefore
entirely removed by a consideration of the laws of information.❞
(Peirce 1866, p. 467)
At this point in his inventory of scientific reasoning, Peirce is
relating the nature of inference, information, and inquiry to the
character of the signs mediating the process in question, a process
he describes as “symbolization”.
In the interest of clarity let's draw from Peirce's account
a couple of quick sketches, designed to show how the examples
he gives of conjunctive terms and disjunctive terms might look
if they were cast within a lattice‑theoretic framework.
Re: Information = Comprehension × Extension • Selection 5
• https://inquiryintoinquiry.com/2024/10/09/information-comprehension-x-exten…
Looking back on Selection 5, let's first examine Peirce's example of a
conjunctive term — “spherical, bright, fragrant, juicy, tropical fruit” —
within a lattice framework. We have the following six terms.
t₁ = spherical
t₂ = bright
t₃ = fragrant
t₄ = juicy
t₅ = tropical
t₆ = fruit
Suppose z is the logical conjunction of the above six terms.
z = t₁ ∙ t₂ ∙ t₃ ∙ t₄ ∙ t₅ ∙ t₆
What on earth could Peirce mean by saying that such a term
is “not a true symbol” or that it is “of no use whatever”?
In particular, consider the following statement.
❝If it occurs in the predicate and something is said
to be a spherical bright fragrant juicy tropical fruit,
since there is nothing which is all this which is not
an orange, we may say that this is an orange at once.❞
(Peirce 1866, p. 470).
In other words, if something x is said to be z then we may guess fairly
surely x is really an orange, in short, x has all the additional features
otherwise summed up quite succinctly in the much more constrained term y,
where y means “an orange”.
Figure 1 shows the implication ordering of logical terms
in the form of a “lattice diagram”.
Figure 1. Conjunctive Term z, Taken as Predicate
• https://inquiryintoinquiry.files.wordpress.com/2016/10/ice-figure-1.jpg
What Peirce is saying about z not being a genuinely useful symbol can
be explained in terms of the gap between the logical conjunction z,
in lattice terms, the greatest lower bound of the conjoined terms,
z = glb{t₁, t₂, t₃, t₄, t₅, t₆}, and what we might regard as the
natural conjunction or natural glb of those terms, namely, y,
“an orange”.
In sum there is an extra measure of constraint which goes into forming the
natural kinds lattice from the free lattice which logic and set theory would
otherwise impose as a default background. The local manifestations of that
global information are meted out over the structure of the natural lattice
by just such abductive gaps as the one we observe between z and y.
Reference —
Peirce, C.S. (1866), “The Logic of Science, or, Induction and Hypothesis”,
Lowell Lectures of 1866, pp. 357–504 in Writings of Charles S. Peirce :
A Chronological Edition, Volume 1, 1857–1866, Peirce Edition Project,
Indiana University Press, Bloomington, IN, 1982.
Resources —
Inquiry Blog • Survey of Pragmatic Semiotic Information
• https://inquiryintoinquiry.com/2024/03/01/survey-of-pragmatic-semiotic-info…
OEIS Wiki • Information = Comprehension × Extension
• https://oeis.org/wiki/Information_%3D_Comprehension_%C3%97_Extension
C.S. Peirce • Upon Logical Comprehension and Extension
• https://peirce.sitehost.iu.edu/writings/v2/w2/w2_06/v2_06.htm
Regards,
Jon
cc: https://www.academia.edu/community/V91eDe