The QwQ system combines LLM technology with traditional AI methods to do the evaluation. This is a hybrid technique that our Permion.ai system uses.
I don't know anything more that I read in the in the following text and the link to a more detailed article. But I believe that hybrid methods are essential for developing reliable and trustworthy AI systems.
John
----------------------
QwQ-32B is an experimental AI model designed to approach problem-solving with deep introspection, emphasizing questioning and reflection before reaching conclusions. Despite its limitations, including language-switching issues and recursive reasoning loops, QwQ demonstrates impressive capabilities in areas like mathematics and coding. For AI practitioners, QwQ represents an attempt to embed a philosophical dimension into reasoning processes, striving for deeper and more robust outcomes—important for teams aiming to build AI that is both effective and adaptable.
QwQ: Reflect Deeply on the Boundaries of the Unknown
https://qwenlm.github.io/blog/qwq-32b-preview
What does it mean to think, to question, to understand? These are the deep waters that QwQ (Qwen with Questions) wades into. Like an eternal student of wisdom, it approaches every problem - be it mathematics, code, or knowledge of our world - with genuine wonder and doubt. QwQ embodies that ancient philosophical spirit: it knows that it knows nothing, and that’s precisely what drives its curiosity. Before settling on any answer, it turns inward, questioning its own assumptions, exploring different paths of thought, always seeking deeper truth. Yet, like all seekers of wisdom, QwQ has its limitations. This version is but an early step on a longer journey - a student still learning to walk the path of reasoning. Its thoughts sometimes wander, its answers aren’t always complete, and its wisdom is still growing. But isn’t that the beauty of true learning? To be both capable and humble, knowledgeable yet always questioning? We invite you to explore alongside QwQ, embracing both its insights and its imperfections as part of the endless quest for understanding.
Limitations
QwQ-32B-Preview is an experimental research model developed by the Qwen Team, focused on advancing AI reasoning capabilities. As a preview release, it demonstrates promising analytical abilities while having several important limitations:
- Language Mixing and Code-Switching: The model may mix languages or switch between them unexpectedly, affecting response clarity.
- Recursive Reasoning Loops: The model may enter circular reasoning patterns, leading to lengthy responses without a conclusive answer.
- Safety and Ethical Considerations: The model requires enhanced safety measures to ensure reliable and secure performance, and users should exercise caution when deploying it.
- Performance and Benchmark Limitations: The model excels in math and coding but has room for improvement in other areas, such as common sense reasoning and nuanced language understanding.
Performance
Through deep exploration and countless trials, we discovered something profound: when given time to ponder, to question, and to reflect, the model’s understanding of mathematics and programming blossoms like a flower opening to the sun. Just as a student grows wiser by carefully examining their work and learning from mistakes, our model achieves deeper insight through patient, thoughtful analysis. This process of careful reflection and self-questioning leads to remarkable breakthroughs in solving complex problems. Our journey of discovery revealed the model’s exceptional ability to tackle some of the most challenging problems in mathematics and programming, including:
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark, a challenging benchmark for evaluating scientific problem-solving abilities through grade school level questions.
- AIME: American Invitation Mathematics Evaluation, which tests mathematical problem solving with arithmetic, algebra, counting, geometry, number theory, and probability and other secondary school math topics.
- MATH-500: The 500 test cases of the MATH benchmark, a comprehensive dataset testing mathematical problem-solving.
- LiveCodeBench: A challenging benchmark for evaluating code generation and problem solving abilities in real-world programming scenarios.
Mike and Igor,
I'm glad that you mentioned design patterns. But instead of an ontology of design patterns, I would say that design patterns should be the BASIS for ontology. In fact, I would point out that visualization and conceptualization are the basis for mathematics. The formal notation always comes at the end, never at the beginning.
The term 'design pattern' is for the kinds of visualizations that mathematicians and logicians start with. The formal notations are essential, but creative mathematicians always start with visual patterns long before they get down to the formal details. In fact, mathematicians usually have the problem completely solved in diagrams long before they work out the formal notations.
The formalism is essential to verify that the visualization is correct and to specify every last detail. And the task of writing out the formal details can often point out issues that were missing or mistaken in the visualization. Visualization is the essence of mathematics. Formalization is the housekeeping that is important for keeping everything neat and tidy.
More generally, I would emphasize the trio of Conceptualization, Analysis, and Formalization. We need all three. For comments by Paul Halmos (a former president of the American Mathematical Society) and Albert Einstein, see the following slide 27. For more discussion and references, see https://jfsowa.com/talks/eswc.pdf .
----------------------------------------
From: "Mike Bennett" <mbennett(a)hypercube.co.uk>
A design pattern ontology would be a very different thing to an ontology
design pattern, but both are things of value.
At the Object Management Group (OMG) we maintain a suite of standards
based on many of these design patterns, i.e. UML and the underpinnings
in MOF.
We are also exploring whether or how to move some of these modeling
languages from being MOF-based to using the new more semantically rich
framework that has been developed for the SysML V2 standards (which has
a kernel language called KerML). These are still in the Finalization
process.
This is where the distinction between model semantics and the semantics
of the target problem domain subject matter become an important
consideration. For example MOF was all about model element semantics.
Ideally some of these directions will move things towards something with
clear formal semantics both for model semantics and how subject matter
semantics is treated. Whether that's in KerML or a more conventional
ontology standard such as FOL or DL, or a syntax such as RDF/OWL,
remains to be seen.
If anyone did happen to be doing a formal ontology of these software
design patterns, this would be very helpful to know.
Meanwhile you should probably also check out the OMG's ESSENCE standard
(spearheaded by Ivar Jacobson) for the kind of model concepts needed to
model a design methodology.
Mike
On 11/25/2024 7:41 PM, 'Igor Toujilov' via ontolog-forum wrote:
> Hi All,
> I am studying a famous book [1] written by the Gang of Four. I am
> surprised that despite it being written 30 years ago, I did not find a
> design pattern ontology. There is plenty of material on ontology
> design patterns on the Internet, but nothing about a design pattern
> ontology which I miss and want to create if it does not exist yet.
> Please advise if I overlooked something.
>
> Regards,
> Igor
>
> [1] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design
> patterns : elements of reusable object-oriented software. 1994.
Differential Logic • 1
• https://inquiryintoinquiry.com/2024/10/30/differential-logic-1-a/
Introduction —
Differential logic is the component of logic whose object is
the description of variation — focusing on the aspects of change,
difference, distribution, and diversity — in universes of discourse
subject to logical description. A definition that broad naturally
incorporates any study of variation by way of mathematical models,
but differential logic is especially charged with the qualitative
aspects of variation pervading or preceding quantitative models.
To the extent a logical inquiry makes use of a formal system, its
differential component governs the use of a “differential logical
calculus”, that is, a formal system with the expressive capacity
to describe change and diversity in logical universes of discourse.
Simple examples of differential logical calculi are furnished by
“differential propositional calculi”. A differential propositional
calculus is a propositional calculus extended by a set of terms for
describing aspects of change and difference, for example, processes
taking place in a universe of discourse or transformations mapping
a source universe to a target universe. Such a calculus augments
ordinary propositional calculus in the same way the differential
calculus of Leibniz and Newton augments the analytic geometry of
Descartes.
Resources —
Survey of Differential Logic
• https://inquiryintoinquiry.com/2024/02/25/survey-of-differential-logic-7/
Regards,
Jon
cc: https://www.academia.edu/community/lJX2qa
cc: https://www.researchgate.net/post/Differential_Logic_The_Logic_of_Change_an…
Pascal,
I just read your paper (cited below). I agree that LLM technology is good for finding important and valuable information. But as you know, there are serious issues about evaluating that information to avoid irrelevant, erroneous, or even hallucinogenic data. I didn't see much attention devoted to evaluation and testing.
As I often mention, our old VivoMind company was doing large volumes of high-speed knowledge extraction, analysis, evaluation, and processing over 20 years ago. For a description of that system with some examples of large applications, see https://jfsowa.com/tallks/cogmem.pdf . The systems described there are just a small sample of the applications, since our customers do not want their data or methods publicized.
I also noticed that you are using OWL for ontology. We use a high-speed version of Prolog, which is much richer, more powerful, and faster than OWL, which implements a tiny subset of the logic that Tim Berners-Lee had proposed for the Semantic Web.
Some of our customers were among the sponsors of the IKRIS project, funded from 2004 to 2006, to support a much larger and more powerful version of what Tim BL had proposed. For an overview of IKRIS with links to some of the original publications, see https://jfsowa.com/ikl .
The IKL technology does not replace LLM, but it is valuable for evaluating the results generated by LLM, detecting errors and avoiding irrelevant, erroneous, or even hallucinogenic data. When processing high volumes of data at high speed, human checking is not possible. High quality computer checking is necessary to eliminate 99% or more of the bad or even dangerous data.
Human checking would only be required for the tiny percentage of data for which the computational methods are uncertain. For a more recent talk, see https://jfsowa.com/talks/eswc.pdf .
John
----------------------------------------
From: "Pascal Hitzler' via ontolog-forum" <ontolog-forum(a)googlegroups.com>
Given the currently ongoing ISWC2024 conference and all the discussions around this neurosymbolic topic: Link to our (with Cogan Shimizu) position paper on this: https://kastle-lab.github.io/assets/publications/2024-LLMs4KGOE.pdf
The developments are really exciting!
Pascal.
Interesting question.
John
____________________
There may be a cosmic speed limit on how fast anything can grow
Alan Turing's theories about computation seem to have a startling consequence, placing hard limits on how fast or slow any physical process in the universe can grow
https://www.newscientist.com/article/2454024-there-may-be-a-cosmic-speed-li…
A newly proposed cosmic speed limit may constrain how fast anything in the universe can grow. Its existence follows from Alan Turing’s pioneering work on theoretical computer science, which opens the intriguing possibility that the structure of the universe is fundamentally linked to the nature of computation.
Cosmic limits aren’t a new idea. While studying the relationship between space and time, Albert Einstein showed that nothing in the universe can exceed the speed of light, as part of his special theory of relativity. Now, Toby Ord at the University of Oxford is proposing a new physical limit based on computation.
“I had the seed of this idea more than 20 years ago,” he says. “It would apply to any quantity you can directly measure, including mass, charge, energy, etc., and even more subtle things like the time intervals between a sequence of events.” . . .
Information = Comprehension × Extension • Comment 1
• https://inquiryintoinquiry.com/2024/10/11/information-comprehension-x-exten…
Re: Information = Comprehension × Extension • Selection 1
• https://inquiryintoinquiry.com/2024/10/05/information-comprehension-x-exten…
All,
Selection 1 ends with Peirce drawing the following conclusion about the
links between information, comprehension, inference, and symbolization.
❝Thus information measures the superfluous comprehension.
And, hence, whenever we make a symbol to express any thing
or any attribute we cannot make it so empty that it shall
have no superfluous comprehension.
❝I am going, next, to show that inference is symbolization
and that the puzzle of the validity of scientific inference
lies merely in this superfluous comprehension and is therefore
entirely removed by a consideration of the laws of information.❞
(Peirce 1866, p. 467)
At this point in his inventory of scientific reasoning, Peirce is
relating the nature of inference, information, and inquiry to the
character of the signs mediating the process in question, a process
he describes as “symbolization”.
In the interest of clarity let's draw from Peirce's account
a couple of quick sketches, designed to show how the examples
he gives of conjunctive terms and disjunctive terms might look
if they were cast within a lattice‑theoretic framework.
Re: Information = Comprehension × Extension • Selection 5
• https://inquiryintoinquiry.com/2024/10/09/information-comprehension-x-exten…
Looking back on Selection 5, let's first examine Peirce's example of a
conjunctive term — “spherical, bright, fragrant, juicy, tropical fruit” —
within a lattice framework. We have the following six terms.
t₁ = spherical
t₂ = bright
t₃ = fragrant
t₄ = juicy
t₅ = tropical
t₆ = fruit
Suppose z is the logical conjunction of the above six terms.
z = t₁ ∙ t₂ ∙ t₃ ∙ t₄ ∙ t₅ ∙ t₆
What on earth could Peirce mean by saying that such a term
is “not a true symbol” or that it is “of no use whatever”?
In particular, consider the following statement.
❝If it occurs in the predicate and something is said
to be a spherical bright fragrant juicy tropical fruit,
since there is nothing which is all this which is not
an orange, we may say that this is an orange at once.❞
(Peirce 1866, p. 470).
In other words, if something x is said to be z then we may guess fairly
surely x is really an orange, in short, x has all the additional features
otherwise summed up quite succinctly in the much more constrained term y,
where y means “an orange”.
Figure 1 shows the implication ordering of logical terms
in the form of a “lattice diagram”.
Figure 1. Conjunctive Term z, Taken as Predicate
• https://inquiryintoinquiry.files.wordpress.com/2016/10/ice-figure-1.jpg
What Peirce is saying about z not being a genuinely useful symbol can
be explained in terms of the gap between the logical conjunction z,
in lattice terms, the greatest lower bound of the conjoined terms,
z = glb{t₁, t₂, t₃, t₄, t₅, t₆}, and what we might regard as the
natural conjunction or natural glb of those terms, namely, y,
“an orange”.
In sum there is an extra measure of constraint which goes into forming the
natural kinds lattice from the free lattice which logic and set theory would
otherwise impose as a default background. The local manifestations of that
global information are meted out over the structure of the natural lattice
by just such abductive gaps as the one we observe between z and y.
Reference —
Peirce, C.S. (1866), “The Logic of Science, or, Induction and Hypothesis”,
Lowell Lectures of 1866, pp. 357–504 in Writings of Charles S. Peirce :
A Chronological Edition, Volume 1, 1857–1866, Peirce Edition Project,
Indiana University Press, Bloomington, IN, 1982.
Resources —
Inquiry Blog • Survey of Pragmatic Semiotic Information
• https://inquiryintoinquiry.com/2024/03/01/survey-of-pragmatic-semiotic-info…
OEIS Wiki • Information = Comprehension × Extension
• https://oeis.org/wiki/Information_%3D_Comprehension_%C3%97_Extension
C.S. Peirce • Upon Logical Comprehension and Extension
• https://peirce.sitehost.iu.edu/writings/v2/w2/w2_06/v2_06.htm
Regards,
Jon
cc: https://www.academia.edu/community/V91eDe
Simon,
I just wanted to add a note about history. From the Wikipedia page about Klaus Tschira: "After gaining his Diplom in physics and working at IBM, Tschira co-founded the German software giant SAP AG in 1972 in Mannheim, Germany."
While he was at IBM, he read a copy of my Conceptual Structures book. He didn't use conceptual graphs, but he applied some of the topics in the book to the technology he used to found SAP. He later invited me to visit the research center he founded in Heidelberg. I recommended some issues about ontology, which led him to organize an invited conference on ontology at his center..
Various participants in that conference later subscribed to Ontolog Forum and worked on ISO standards projects for Common Logic and ontology.. This is one of many reasons why the 60+ years of symbolic AI is ESSENTIAL for practical and non-hallucinogenic applications of LLMs.
John
----------------------------------------
From: "Polovina, Simon (BTE)' via ontolog-forum" <ontolog-forum(a)googlegroups.com>
Hi all!
Connecting the Facts: SAP HANA Cloud’s Knowledge G... - SAP Community may interest you. Combining knowledge graphs with LLM as a commercial product by the business computing market leader, SAP.
Simon
Dr Simon Polovina
Department of Computing, Sheffield Hallam University
Gary,
The word 'limits" sounds negative. That is why I recommend a positive way of describing the distinction of continuous and discrete representations.
1. The mapping to and from the world depends of continuous representations, such as differential equations and an ope-ended variety of technologies for mapping information about the world for many kinds of applications.
2. Various kinds of diagrams and graphs represent discrete models of the world and things in it. Mappings of those representations to and from computable forms may use various kinds of formal logics.
3. Natural languages can represent anything that humans experience, think, plan, do, or intend to do. Therefore, they should be able to represent anything and everything in #1 and #2.at any level of precision or approximation.
The top two lines describe the continuous and discrete. The third line shows how and why people can describe both sides while using their native language. An important point about using NLs for discussing formal topics: : Every textbook on logic defines the subject by sentences in some NL. Therefore it is possible (but not easy) to talk precisely about formal methods.
However, it is too easy to slip into vagueness. The exercise of mapping NLs to a forma logic forces humans to be absolutely precise. But many people don't know how to do that. Therefore, it's important to develop interactive tools to aid in the translation.
John.
----------------------------------------
From: "Gary Berg-Cross" <gbergcross(a)gmail.com>
Sent: 10/18/24 1:48 PM
To: ontolog-trustee(a)googlegroups.com
Cc: ontolog-forum <ontolog-forum(a)googlegroups.com>, CG <cg(a)lists.iccs-conference.org>
Subject: Re: [ontolog-trustee] Trying to develop a proper useful topic for the 2025 summit
With John's points as background I suggest that the way to frame a workable summit topic would be to explore the current and likely limits to useful formalization.
Gary Berg-Cross
Potomac, MD
240-426-0770