An end-to-end deep learning project that generates natural-language captions for images using a pretrained Xception model and an LSTM-based decoder.
The complete workflow was developed in Google Colab using the Flickr8k dataset.
This project processes image-caption pairs from the Flickr8k dataset and trains a deep learning model to generate captions for unseen images.
The system uses:
- Xception for extracting visual features from images
- An Embedding layer for representing caption sequences
- LSTM for processing text sequences
- A combined decoder network for predicting the next word in a caption
- Loads and cleans image captions
- Builds a vocabulary from the training captions
- Adds
startseqandendseqtokens to captions - Extracts image features using pretrained Xception
- Stores extracted features in an HDF5 file
- Creates and saves a tokenizer
- Generates training sequences in batches
- Trains a CNN-LSTM caption-generation model
- Uses model checkpointing and early stopping
- Generates captions for test images and custom images
- Saves the final trained model
- Python
- TensorFlow
- Keras
- NumPy
- Pandas
- Matplotlib
- Pillow
- HDF5
- Google Colab
The project uses the Flickr8k Dataset, which contains images paired with multiple human-written captions.
The notebook uses the following files:
Flicker8k_Dataset/
Flicker8k_Text/
├── Flickr8k.token.txt
├── Flickr_8k.trainImages.txt
└── Flickr_8k.testImages.txt
The dataset is not included in this repository because of its size.
The model accepts two inputs.
- Images are resized to
299 × 299 - Features are extracted using Xception pretrained on ImageNet
- Each image is represented using a
2048-dimensional feature vector - Dropout and a dense layer are applied to the extracted features
- Captions are converted into numerical sequences
- Sequences are padded to the maximum caption length
- An Embedding layer creates word representations
- An LSTM layer processes the caption sequence
The image and caption representations are combined and passed through dense layers to predict the next word.
- Optimizer: Adam
- Loss Function: Categorical Cross-Entropy
- Label Smoothing:
0.1 - Batch Size:
64 - Maximum Epochs:
20 - Image Feature Size:
2048 - Embedding Dimension:
256 - LSTM Units:
256 - Model Checkpointing: Enabled
- Early Stopping: Enabled
Image-Captioning-System/
├── Image_Captioning.ipynb
├── README.md
└── requirements.txt
Clone the repository:
git clone https://github.com/bhatiaanmol/Image-Captioning-System.git
cd Image-Captioning-SystemInstall the required libraries:
pip install -r requirements.txt- Open
Image_Captioning.ipynbin Google Colab. - Mount Google Drive.
- Place the Flickr8k dataset inside the required Google Drive directory.
- Update the dataset paths in the notebook if necessary.
- Run the cells in sequence.
- Extract image features or load the existing
features.h5file. - Train the model or load weights from
best_model.h5. - Provide a test image to generate a caption.
The notebook generates the following files during execution:
descriptions.txt
features.h5
tokenizer.pkl
best_model.h5
model.h5
caption_model_final.keras
These files are not included in the repository because they can be regenerated by running the notebook and some may be large.
The trained model begins with the token startseq and predicts one word at a time until it predicts endseq or reaches the maximum caption length.
The notebook supports caption generation for:
- Images from the Flickr8k test set
- Custom images provided by the user
The model generates a caption for a custom image.
Generated Caption: man in blue shirt is standing on the beach
- Add beam search for improved caption generation
- Evaluate the model using BLEU scores
- Add an attention mechanism
- Build a Streamlit interface
- Experiment with transformer-based captioning models
Anmol Bhatia
- GitHub: bhatiaanmol
- LinkedIn: Anmol Bhatia