Large-Scale Coarse-to-Fine Object Retrieval Ontology and Deep Local Multitask Learning

Algorithm 1

Offline-phase CFOR system algorithm applied in the fashion field.

	Procedure: OFFLINE_CFOR
	Goals:
	Establish fashion ontology through general ontology and predicates extracted from data.
	Train regions, category classification models, and attribute multitask classification models
	Establish inverted index files for retrieval
	Establish database which contains all extracted features and information of all objects in all images in data
	Input:
	dta(obj) //image object database, i.e., fashion object detected by the inherited object detector for all images in data.
	prior(field) //structured information (predicates) which contains semantic concepts, attributes, and their correlation in a specific field (here, we have fashion field); for example, Blazer is a category.
	Output:
	onto(prior, dta) //established ontology based on prior knowledge, i.e., general ontology structure
	classifyModel(state, onto, dta) //a trained classification model in a specific state will be used for online phase (retrieval phase). Through experimental results, ResNet architecture is suitable to apply.
	States //including concept state (region or category) and attribute state (color, shape, part, style, etc.)
	multitaskModel(state, onto, dta) //a trained multitask classification model for a specific state belonging to the attribute state. Through experimental results, NasNet architecture is suitable to apply.
	indexFiles //inverted index files, i.e., for speeding up retrieval time
	Database //database contains all extracted features and information of all objects in all images in data.
	BEGIN
	//Ontology establishment
	predicates ⟵ extract_predicates(dta)  //extract concepts, attributes, and their correlation in fashion data to generate predicates
	onto ⟵ build_ontology(predicates, prior)  //integrate predicates into a general form of ontology to build up a fashion ontology
	//Training phase
	states ⟵ extract_state(onto)  //extract states of ontology including concepts (region and category) and attributes (color, part, shape, style, texture, and fabric)
	group_state_dta ⟵ NULL  //store all attribute state data for imbalanced data problem solver
	for state in states:  //build up classification model for each ontology state (except leaf states in ontology)
	begin
	if state in concepts:
	begin
	//extract necessary data for training model in current state of ontology instead of using all data for training
	state_dta ⟵ extract_nes_dta(dta, state onto)
	//train classification model for current state whose architecture is chosen from NASNet/ResNet indicated by ontology
	classifyModel(architecture, state_dta)
	end
	if state in attributes:
	begin
	//extract attribute data for training and store in a group for imbalanced data problem solver
	state_dta ⟵ extract_nes_dta(dta, state, onto)
	group_state_dta.append(state_dta, onto)
	//train attribute multitask classification model with Matthews’ correlation coefficient-based imbalanced data problem solver
	multitaskModel(group_state_dta, architecture, Matthrew_coef = True)
	End
	end
	//Indexing
	indexFiles ⟵ NULL    //store all index files based on the operation of ontology
	//make inverted index file
	for state in states:
	begin
	state_dt ⟵ extracted_nes_dta(dta, state, onto)
	cur_indexFile ⟵ indexing(state_sta)
	indexFiles.append(cur_indexFile)
	end
	//Build a storing structure for semantic information and extracted features
	storage ⟵ build_storage(onto, states)
	//extract all semantic information from input objects
	infor_dta ⟵ infor_extract(states, dta, onto, classifyModels, multitaskModel)
	//global and local object feature extraction
	feat_dta ⟵ feat_extract(dta, onto, classificationModels, multitaskModel)
	//construct storing database for all extracted information
	database ⟵ structure(storage, feat_dta, info_dta, indexFiles)
	END