At Fractal Labs, we’ve taken on the challenge of developing a predictive algorithm to power a new investment tool, currently called Your Personal Quant. Designed for day traders, this app uses advanced machine learning to predict stock movements over short timeframes, combining historical market data with real-time sentiment analysis from news articles. Unlike traditional AI projects where we rely on pre-built models, this project marked our first experience building a custom machine learning model from scratch.

In this article we’ll dive into how our predictive algorithm works, the steps we’ve taken to ensure its accuracy, and how this technology can be applied to other projects.

How the Algorithm Works

The model processes time-series data to identify patterns in stock price movements. Specifically, it looks at data points such as:

• Daily closing prices: The final price of a stock at the close of each trading day.
• Price fluctuations: How much a stock’s price changes over time.
• Volume: The number of shares traded over a certain period.

The model analyzes these historical trends to generate forecasts of how a stock’s price might change in the near future, typically over the next day or the following few days. For instance, if a user is monitoring a stock like Apple, the algorithm uses Apple’s past price movements to predict where the stock price might go, allowing day traders to make informed buy, sell, or hold decisions.

Custom Machine Learning Model

One of the most exciting aspects of this project was that we built the machine learning model from the ground up, using TensorFlow. Many of our previous AI projects didn’t require the level of customization and training as this project, so we used external APIs and some model training techniques to deliver the outcome. However, this project required us to take control of the entire pipeline—from cleaning the data to training the model.

This included:

• Data cleaning: Removing inconsistencies or outliers in the historical stock data to avoid skewed predictions.
• Feature selection: Choosing which data points, such as price and volume, were most relevant for making accurate predictions.
• Model training: Feeding the processed data into the machine learning model and refining it through multiple iterations to improve its prediction accuracy.

By building our own model, we were able to tailor it specifically to the needs of day traders, ensuring that the algorithm focuses on the short-term movements that are critical for high-frequency trading.

Adding Sentiment Analysis: A Layer of Qualitative Data

Quantitative data from the stock market tells only part of the story. Stock prices can also be influenced by market sentiment, particularly how traders and the public react to news events. To account for this, we integrated sentiment analysis into the predictive model.

How Sentiment Analysis Enhances Stock Predictions

Our sentiment analysis feature pulls in relevant news articles about the stock and applies natural language processing (NLP) to determine whether the news is generally positive, negative, or neutral. This information adds a qualitative layer to the algorithm, helping to explain market fluctuations that might not be captured by historical data alone.

For example, if Apple releases a new product or reports strong earnings, positive sentiment in the news can push the stock price higher, even if historical trends alone wouldn’t predict such a rise. Conversely, negative coverage—such as legal troubles or product delays—might signal a decline in stock price, regardless of historical performance.

Fine-Tuning for Financial Sentiment

To ensure that the sentiment analysis is as accurate as possible, we used a technique called fine-tuning. Rather than building the sentiment model from scratch, we adapted existing large language models (LLMs) like GPT or Claude by training them on a dataset of financial news. Fine-tuning allowed us to specialize these models for the specific language and jargon used in stock market reporting.

The fine-tuning process helped us make the model more attuned to subtle nuances in financial sentiment. For instance, a general-purpose language model might interpret an earnings report as neutral, but our fine-tuned model understands that certain phrases in financial news, such as "lower-than-expected earnings," may indicate negative sentiment for stock traders.

Combining Data for a Holistic Stock Forecast

The strength of the predictive algorithm lies in how it combines both quantitative predictions (from historical data) and qualitative sentiment analysis (from news articles). The model weighs both data sources to provide a more well-rounded prediction for traders.

Here’s how it works:

• Short-term stock forecasts: Based on historical data, the model predicts how the stock is likely to behave in the next day or few days.
• Sentiment scores: The sentiment analysis gives a score indicating whether the news coverage around the stock is positive, negative, or neutral.
• Combined prediction: The model then generates a single buy, sell, or hold recommendation, along with a confidence score that reflects the model’s certainty in its forecast.

This dual-layer approach ensures that traders are not only informed by historical trends but also aware of how current events might impact stock prices in the immediate future. By integrating real-time sentiment analysis, we provide users with an accurate, up-to-date prediction of stock performance.

Ensuring the Accuracy of Predictions

Here are the steps we take to ensure that the predictive algorithm delivers reliable and actionable insights to users:

1. Continuous Model Training

The stock market is constantly evolving, and a model that was accurate last year may not be as effective today. To ensure the predictions stay relevant, we continuously update the model with new data, allowing it to learn from recent trends and market shifts.

2. Data Validation and Cleaning

A critical aspect of model accuracy is the quality of the data used. Before feeding any data into the model, we go through a rigorous data cleaning process to remove any anomalies or errors that could distort the predictions.

3. Hyperparameter Tuning

During development, we used hyperparameter tuning to optimize the model’s performance. By adjusting key settings within the model, such as learning rates and batch sizes, we ensured that the algorithm was configured to deliver the highest possible accuracy.

4. Cross-Validation

We implemented cross-validation techniques to make sure that the model performs well on unseen data. This helps prevent overfitting, which can occur when a model becomes too specialized in its training data and struggles to generalize in real-world scenarios.

5. Confidence Scores

Each prediction is delivered with a confidence score, which indicates how certain the model is about its forecast. This allows users to weigh the reliability of the prediction and make informed decisions about how much risk they’re willing to take.

Applying Our Predictive Algorithm Expertise to Other Projects

The development of Your Personal Quant provided us with valuable expertise in building custom machine learning models, fine-tuning large language models, and integrating sentiment analysis—skills that are adaptable across various industries. The custom predictive model we developed for analyzing stock data can easily be applied to any field involving time-series data, such as predicting patient outcomes in healthcare, forecasting supply chain fluctuations in logistics, or analyzing market trends in retail.

Additionally, our work with sentiment analysis offers valuable real-time insights that extend beyond finance. For example, we can use sentiment analysis in marketing to gauge public opinion on a brand or product, in legal tech to assess the impact of court rulings, or in customer service to monitor feedback and improve product development. Our expertise in fine-tuning models for specific language and industry needs allows us to deliver tailored AI solutions that adapt to the unique challenges of each domain.

Expertise in Predictive Algorithms and Real-Time Data Analysis

The development of the predictive algorithm behind Your Personal Quant has provided us with deep expertise in building custom machine learning models, fine-tuning them for specific applications, and integrating sentiment analysis for real-time insights. Our ability to combine quantitative market data with qualitative sentiment analysis delivers powerful stock market predictions that empower day traders with actionable insights.

This expertise is not limited to stock market applications. The skills and technologies we’ve developed can be applied to other industries, enabling us to deliver highly specialized predictive tools for clients across various sectors.

‍