How the AI works, and where it can be wrong

Effective: 2026-05-15 · Version 2026-05-15

SignalSnitch uses large language models (LLMs) to read posts from public social-media accounts and turn them into structured data — a ticker, an action (BUY or SELL), a confidence label. Those structured signals are what feed the accuracy stats, the consensus detector, and the signal cards you see in the dashboard. LLMs make mistakes. This page exists so you know where, and what we do about it.

1. What "AI" means here

Every post we scrape passes through an LLM-based analyzer. The analyzer's job is to answer three questions about each post:

Is this post making a trading call at all, or is it commentary, a chart, a meme, news?
If it is a call, what ticker (or tickers) is it about?
What direction — BUY/long or SELL/short — and how confident does the caller sound?

Those structured outputs are then stored in our database and used to compute everything downstream. If the model misreads a post, the error propagates through every one of those numbers.

2. Which models we use

SignalSnitch sends post text to multiple LLMs, including Anthropic Claude and OpenAI GPT. We send only the public post text — never your account data, your watchlist, your billing info, or any other private state.

3. Failure modes we've actually seen

3.1 Ticker hallucination

The model invents a ticker that doesn't appear anywhere in the source post. We run a verification step that drops signals whose ticker doesn't literally appear in the source text, which catches the common cases.

3.2 Sentiment misread

The model flips BUY/SELL on a post that uses irony, hedging, or a confusing structure. We rely on the consensus filter (≥2 callers agreeing) to dilute the impact of a single misread.

3.3 Non-call classified as a call

Commentary or a screenshot caption that mentions a ticker gets promoted to a real BUY/SELL signal. We use the "Non-Calls" chip on the influencer detail page so you can audit what the classifier did and didn't flag.

3.4 Time-zone / date misread

The model can occasionally misread the posting time relative to market open, which affects the "entry price" we capture.

4. What we measure

A caller's accuracy is based on what actually happened in the market after each of their calls — not on how confident the AI thought the caller sounded. We wait long enough after a call before scoring it so the verdict reflects real movement rather than first-minute noise.

Sample size matters more than the headline number. We always show the number of calls next to every percentage so you can weigh it yourself.

5. Safeguards we run

We filter out signals where the AI invented a ticker that wasn't actually in the post. We don't treat a single caller's opinion as a "consensus" — several independent callers have to agree first. We score wins and losses from real price data after the fact. And we periodically re-check older signals as our analysis improves.

6. What you should do

Click through to the source post. Every signal card links to the original tweet / Reddit thread / StockTwits message. Read the post yourself before acting on any signal.

Treat numbers as ranges, not points. A "65% accuracy" figure has uncertainty bands around it from sample size, analyzer error, and survivorship.

Report mistakes you spot. Email support@signalsnitch.io with the signal's URL. We use those reports to refine the analyzer prompt.

7. Bottom line

SignalSnitch is a research tool that uses imperfect AI on imperfect input (social media) to produce imperfect output. The pipeline is good enough to surface patterns and study crowd psychology — which is the use case we promise. It is not good enough to be a trading signal you act on without your own verification.