The QwQ system combines LLM technology with traditional AI methods to do the evaluation. This is a hybrid technique that our Permion.ai system uses.
I don't know anything more that I read in the in the following text and the link to a more detailed article. But I believe that hybrid methods are essential for developing reliable and trustworthy AI systems.
John
----------------------
QwQ-32B is an experimental AI model designed to approach problem-solving with deep introspection, emphasizing questioning and reflection before reaching conclusions. Despite its limitations, including language-switching issues and recursive reasoning loops, QwQ demonstrates impressive capabilities in areas like mathematics and coding. For AI practitioners, QwQ represents an attempt to embed a philosophical dimension into reasoning processes, striving for deeper and more robust outcomes—important for teams aiming to build AI that is both effective and adaptable.
QwQ: Reflect Deeply on the Boundaries of the Unknown
https://qwenlm.github.io/blog/qwq-32b-preview
What does it mean to think, to question, to understand? These are the deep waters that QwQ (Qwen with Questions) wades into. Like an eternal student of wisdom, it approaches every problem - be it mathematics, code, or knowledge of our world - with genuine wonder and doubt. QwQ embodies that ancient philosophical spirit: it knows that it knows nothing, and that’s precisely what drives its curiosity. Before settling on any answer, it turns inward, questioning its own assumptions, exploring different paths of thought, always seeking deeper truth. Yet, like all seekers of wisdom, QwQ has its limitations. This version is but an early step on a longer journey - a student still learning to walk the path of reasoning. Its thoughts sometimes wander, its answers aren’t always complete, and its wisdom is still growing. But isn’t that the beauty of true learning? To be both capable and humble, knowledgeable yet always questioning? We invite you to explore alongside QwQ, embracing both its insights and its imperfections as part of the endless quest for understanding.
Limitations
QwQ-32B-Preview is an experimental research model developed by the Qwen Team, focused on advancing AI reasoning capabilities. As a preview release, it demonstrates promising analytical abilities while having several important limitations:
- Language Mixing and Code-Switching: The model may mix languages or switch between them unexpectedly, affecting response clarity.
- Recursive Reasoning Loops: The model may enter circular reasoning patterns, leading to lengthy responses without a conclusive answer.
- Safety and Ethical Considerations: The model requires enhanced safety measures to ensure reliable and secure performance, and users should exercise caution when deploying it.
- Performance and Benchmark Limitations: The model excels in math and coding but has room for improvement in other areas, such as common sense reasoning and nuanced language understanding.
Performance
Through deep exploration and countless trials, we discovered something profound: when given time to ponder, to question, and to reflect, the model’s understanding of mathematics and programming blossoms like a flower opening to the sun. Just as a student grows wiser by carefully examining their work and learning from mistakes, our model achieves deeper insight through patient, thoughtful analysis. This process of careful reflection and self-questioning leads to remarkable breakthroughs in solving complex problems. Our journey of discovery revealed the model’s exceptional ability to tackle some of the most challenging problems in mathematics and programming, including:
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark, a challenging benchmark for evaluating scientific problem-solving abilities through grade school level questions.
- AIME: American Invitation Mathematics Evaluation, which tests mathematical problem solving with arithmetic, algebra, counting, geometry, number theory, and probability and other secondary school math topics.
- MATH-500: The 500 test cases of the MATH benchmark, a comprehensive dataset testing mathematical problem-solving.
- LiveCodeBench: A challenging benchmark for evaluating code generation and problem solving abilities in real-world programming scenarios.
Mike and Igor,
I'm glad that you mentioned design patterns. But instead of an ontology of design patterns, I would say that design patterns should be the BASIS for ontology. In fact, I would point out that visualization and conceptualization are the basis for mathematics. The formal notation always comes at the end, never at the beginning.
The term 'design pattern' is for the kinds of visualizations that mathematicians and logicians start with. The formal notations are essential, but creative mathematicians always start with visual patterns long before they get down to the formal details. In fact, mathematicians usually have the problem completely solved in diagrams long before they work out the formal notations.
The formalism is essential to verify that the visualization is correct and to specify every last detail. And the task of writing out the formal details can often point out issues that were missing or mistaken in the visualization. Visualization is the essence of mathematics. Formalization is the housekeeping that is important for keeping everything neat and tidy.
More generally, I would emphasize the trio of Conceptualization, Analysis, and Formalization. We need all three. For comments by Paul Halmos (a former president of the American Mathematical Society) and Albert Einstein, see the following slide 27. For more discussion and references, see https://jfsowa.com/talks/eswc.pdf .
----------------------------------------
From: "Mike Bennett" <mbennett(a)hypercube.co.uk>
A design pattern ontology would be a very different thing to an ontology
design pattern, but both are things of value.
At the Object Management Group (OMG) we maintain a suite of standards
based on many of these design patterns, i.e. UML and the underpinnings
in MOF.
We are also exploring whether or how to move some of these modeling
languages from being MOF-based to using the new more semantically rich
framework that has been developed for the SysML V2 standards (which has
a kernel language called KerML). These are still in the Finalization
process.
This is where the distinction between model semantics and the semantics
of the target problem domain subject matter become an important
consideration. For example MOF was all about model element semantics.
Ideally some of these directions will move things towards something with
clear formal semantics both for model semantics and how subject matter
semantics is treated. Whether that's in KerML or a more conventional
ontology standard such as FOL or DL, or a syntax such as RDF/OWL,
remains to be seen.
If anyone did happen to be doing a formal ontology of these software
design patterns, this would be very helpful to know.
Meanwhile you should probably also check out the OMG's ESSENCE standard
(spearheaded by Ivar Jacobson) for the kind of model concepts needed to
model a design methodology.
Mike
On 11/25/2024 7:41 PM, 'Igor Toujilov' via ontolog-forum wrote:
> Hi All,
> I am studying a famous book [1] written by the Gang of Four. I am
> surprised that despite it being written 30 years ago, I did not find a
> design pattern ontology. There is plenty of material on ontology
> design patterns on the Internet, but nothing about a design pattern
> ontology which I miss and want to create if it does not exist yet.
> Please advise if I overlooked something.
>
> Regards,
> Igor
>
> [1] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design
> patterns : elements of reusable object-oriented software. 1994.
Differential Logic • 1
• https://inquiryintoinquiry.com/2024/10/30/differential-logic-1-a/
Introduction —
Differential logic is the component of logic whose object is
the description of variation — focusing on the aspects of change,
difference, distribution, and diversity — in universes of discourse
subject to logical description. A definition that broad naturally
incorporates any study of variation by way of mathematical models,
but differential logic is especially charged with the qualitative
aspects of variation pervading or preceding quantitative models.
To the extent a logical inquiry makes use of a formal system, its
differential component governs the use of a “differential logical
calculus”, that is, a formal system with the expressive capacity
to describe change and diversity in logical universes of discourse.
Simple examples of differential logical calculi are furnished by
“differential propositional calculi”. A differential propositional
calculus is a propositional calculus extended by a set of terms for
describing aspects of change and difference, for example, processes
taking place in a universe of discourse or transformations mapping
a source universe to a target universe. Such a calculus augments
ordinary propositional calculus in the same way the differential
calculus of Leibniz and Newton augments the analytic geometry of
Descartes.
Resources —
Survey of Differential Logic
• https://inquiryintoinquiry.com/2024/02/25/survey-of-differential-logic-7/
Regards,
Jon
cc: https://www.academia.edu/community/lJX2qa
cc: https://www.researchgate.net/post/Differential_Logic_The_Logic_of_Change_an…
Pascal,
I just read your paper (cited below). I agree that LLM technology is good for finding important and valuable information. But as you know, there are serious issues about evaluating that information to avoid irrelevant, erroneous, or even hallucinogenic data. I didn't see much attention devoted to evaluation and testing.
As I often mention, our old VivoMind company was doing large volumes of high-speed knowledge extraction, analysis, evaluation, and processing over 20 years ago. For a description of that system with some examples of large applications, see https://jfsowa.com/tallks/cogmem.pdf . The systems described there are just a small sample of the applications, since our customers do not want their data or methods publicized.
I also noticed that you are using OWL for ontology. We use a high-speed version of Prolog, which is much richer, more powerful, and faster than OWL, which implements a tiny subset of the logic that Tim Berners-Lee had proposed for the Semantic Web.
Some of our customers were among the sponsors of the IKRIS project, funded from 2004 to 2006, to support a much larger and more powerful version of what Tim BL had proposed. For an overview of IKRIS with links to some of the original publications, see https://jfsowa.com/ikl .
The IKL technology does not replace LLM, but it is valuable for evaluating the results generated by LLM, detecting errors and avoiding irrelevant, erroneous, or even hallucinogenic data. When processing high volumes of data at high speed, human checking is not possible. High quality computer checking is necessary to eliminate 99% or more of the bad or even dangerous data.
Human checking would only be required for the tiny percentage of data for which the computational methods are uncertain. For a more recent talk, see https://jfsowa.com/talks/eswc.pdf .
John
----------------------------------------
From: "Pascal Hitzler' via ontolog-forum" <ontolog-forum(a)googlegroups.com>
Given the currently ongoing ISWC2024 conference and all the discussions around this neurosymbolic topic: Link to our (with Cogan Shimizu) position paper on this: https://kastle-lab.github.io/assets/publications/2024-LLMs4KGOE.pdf
The developments are really exciting!
Pascal.
Interesting question.
John
____________________
There may be a cosmic speed limit on how fast anything can grow
Alan Turing's theories about computation seem to have a startling consequence, placing hard limits on how fast or slow any physical process in the universe can grow
https://www.newscientist.com/article/2454024-there-may-be-a-cosmic-speed-li…
A newly proposed cosmic speed limit may constrain how fast anything in the universe can grow. Its existence follows from Alan Turing’s pioneering work on theoretical computer science, which opens the intriguing possibility that the structure of the universe is fundamentally linked to the nature of computation.
Cosmic limits aren’t a new idea. While studying the relationship between space and time, Albert Einstein showed that nothing in the universe can exceed the speed of light, as part of his special theory of relativity. Now, Toby Ord at the University of Oxford is proposing a new physical limit based on computation.
“I had the seed of this idea more than 20 years ago,” he says. “It would apply to any quantity you can directly measure, including mass, charge, energy, etc., and even more subtle things like the time intervals between a sequence of events.” . . .