CNN,LSTM,NLP BASED VIDEO RETRIEVAL SYSTEM

inbotics Machine Learning July 9, 2022 | 0

ABSTRACT:-

Nowadays, there are a lot of people who have a smartphone for taking pictures or images which contain Different objects. Different objects like place, person, etc. But there is no use if someone click picture by smartphone but not able to recognize what image is showing, if person not understand the what image is showing or describing, Image captioning is technique used for this issue. If a user has a disability like blindness or Paralysis, this system may help to understand the image or recognize what is in it. This project is all about real time image captioning used for the video retrieval process. This system is totally based on CNN, LSTM and NLP algorithms. In this project, we trained the Flickr 8k dataset with CNN and LSTM algorithms to create a model which was used to caption the frames of videos. In this dataset ,10 captions are maintained for each image in the dataset and there are 8000 images in the dataset. If we pass video as input to our system, first it will extract all frames from the video. And pass it to the model to obtain a caption for each image and this caption and time will store in MySQL database. If any user searches a query NLP, if similar words are found in the database it will display the video with keyword and time to the user. The very main point in image captioning is to recognize non-identical objects in an image, find what is alike in that and then we need to classify them and merge the words that might not be good in terms of language Modelling. Captions with correct sentences require two important technologies that are Computer Vision and Natural Language Processing for getting correct Sentences. When objects are classified then the objects are passed in the Language Model to Create Options Semantics Knowledge for an Image to get by capturing Characteristics of an image Globally and Locally.

OBJECTIVES OF THE PROJECT:-

The main objectives of this system are:

Absorption the concept of CNN for feature extraction and feature vector. Understanding the visual semantics in the real time image and converting it into a simple caption.
Image captioning is the process of generating natural language explanation according to the content observed in the image which is done by the understanding of computer vision and natural language processing.
To find a person or any object in the video by the help of image captioning and Natural Language Processing.
Understanding the concept of Convolutional neural network (CNN), Long Short-Term Memory (LSTM) and Natural Language Processing (NLP).
System has a capability to generate the proper output for any query of the user.

EXISTING SYSTEM:-

The present system for motion recognition includes the usage of Convolutional Neural Networks (CNN). Videos are taken as a chain of frames and frame-stage CNN series capabilities generated are fed to Long Short-Term Memory (LSTM) version for video recognition. However, the above mentioned method takes frame-level CNN series functions as enter for LSTM, which might also additionally fail to capture the wealthy movement facts from adjoining frames or a couple of clips. It is crucial to take into account adjoining frames that permit for salient functions, rather than mapping a whole body right into a static representation. Thereby, to mitigate this drawback, a brand new method is proposed in which initially, saliency-conscious techniques are carried out to generate saliency-conscious videos. Then, an end-to-end pipeline is designed through integrating 3-D CNN with LSTM, observed through a time collection pooling layer and a softmax layer to predict the sports in video.

PROPOSED SYSTEM:-

Nowdays, many people have smartphones to click an image and capture memory. Images are not only for saving memories, it has a lot of meaningful information to pass to the viewer. This system is extremely helpful for blind people by searching for a query which they want. This system is very helpful according to future objectives. This system is useful for blind people because it displays the timer of a video in which the person or object present which was searched before the user.

In this system , we used 3 algorithms like CNN, LSTM and NLP. CNN has a major use in the image captioning process. CNN and LSTM are combinedly used for the image captioning. In this we used a Flickr 8k dataset which is easily available on the internet. It has 8000 random images with at least 10 captions for each image. In the previous paper, this was done by OCR and feature extraction. But as we know it’s an old way of doing this. I.e we solve this problem by emerging technologies and algorithms. We used ML algorithms such as CNN(Convolutional neural network), LSTM(Long short term memory) and NLP(Natural processing language). We will train our model with flickr 8k dataset and save it according to caption using LSTM. CNN and LSTM are combinedly worked in the captioning process. CNN divides the video into no of frames and these frames will be passed into model to obtain caption for each image. These captions and time are stored in the MySQL database. When a user searches for a query, if the system finds a similar word in the database it will take it and those videos with that keyword and time will display to the user as output.

MODULES:

User Register: User have to register themself to retrieve particular image from

Video.

User Login: User have to register themself to retrieve particular image from

Video.

Text Model Generation: We will use Natural Language Processing (NLP) to analyse the caption and based on that system retrieve the image.
CNN and LSTM Model Generation: We use flickr 8k dataset to train CNN and LSTM model generation.
CNN Model Generation: CNN for feature extraction and feature vector. Understanding the visual semantics in the real time image and converting it into a simple caption.

ADVANTAGES :

With the ML user need not enter each and every step of processing.
User will be able to retrieve all the information stored in video as per his convenience.
The user only sees his queries answered for example white man. Then all the videos where white man comes with a time frame.
80% automated system.

HARDWARE AND SOFTWARE REQUIREMENTS

HARDWARE:-

Processor: i3 ,i5
RAM: 8GB
Hard disk: 16 GB

SOFTWARE:-

operating System : Windows 2000/XP/7/8/10
Anaconda , jupyter , spyder, flask
Frontend :-python
Backend:- MYSQL