In the inaugural edition of IES Generative AI Challenge 2024, we had 150 applicants from 13 countries. Out of them, 43 were shortlisted for the hackathon and 24 teams completed all 3 milestones. Judges selected 12 for the final round, and 10 manuscripts were submitted to IECON 2024 (Track SS41 – The Responsible Practice of Generative Artificial Intelligence for Industrial Applications and Systems).
9 of these papers were accepted and presented at IECON 2024. We supported 6 teams with travel grants to visit Chicago, IL, USA, in November 2024, to attend the conference in person. Below are some photos from the session.
Hackathon outputs
![]() |
Topic: Generative AI and EEG-based Music Personalization for Work Stress Reduction
The escalating prevalence of work-related stress has led to a notable decline in work performance and the mental well-being of the workforce. Studies suggest that personally tailored music can have a positive impact on reducing stress levels. This paper proposes a novel approach to generate personalized music by combining brain activation analysis and musical preference analysis. The proposed method involves three main stages: Source Separation, Brain Activation Analysis, Personalized Music Generation. In the first stage, we use a variant of the Wave-U-Net to decompose the input song into its vocals and melodies. Subsequently, we employ electroencephalography (EEG) to analyze the user's brain activation and identify preferred segments. The third stage involves the generation of personalized music using a transformer-based music generation model, which takes into account the user's music preference and brain activation patterns. Furthermore, this approach was assessed through an experiment with 10 participants, achieving an average satisfaction level of 3.9/5 and a 52.97% increase in Frontal Alpha Asymmetry (FAA) fluctuation percentage after analyzing brain activation with the generated melody. As future work, we intend to conduct intervention-based experiments with large sample sizes and expand into evaluation metrics that provide objective measures of work stress reduction. |
|
![]() |
Topic: Forecasting Extreme Price Fluctuations in Energy Markets using Large Language Models
This work addresses the challenge of accurately forecasting electricity prices within the volatile Australian market, especially during extreme conditions. It leverages advanced generative pre-trained Large Language Models (LLMs) to analyze the content of electricity market notices with the goal of identifying the drivers behind extreme price fluctuations. Additionally, this approach employs LLMs for an in-depth time-series analysis of electricity prices, providing Australian electricity company traders with insights to refine their trading strategies. To enhance forecasting accuracy, this study adopts the QLoRA method for fine-tuning open-access LLMs, enabling the analysis of market notices to generate a time-series event dataset. A CNN-LSTM network architecture is designed to process both electricity price data and market notice information, thereby improving forecast precision in periods of extreme price volatility. The proposed decision support framework undergoes simulation and evaluation using data from the Australian electricity market, demonstrating its potential to significantly benefit traders in navigating the complexities of the energy sector. |
|
![]() |
Topic: Predictive Maintenance with Large Language Models and Transformer-based Survival Analysis
Predictive maintenance involves proactive monitoring and prediction of equipment failures to optimize maintenance schedules to minimize downtime, reduce losses, and ensure optimal equipment performance. However, traditional methods often rely on classification or regression, overlooking the temporal aspect of failure prediction. Survival analysis is focused on modeling time-to-event data, making it well-suited for predicting the time until equipment failures. The primary goal of this paper is to evaluate the feasibility and effectiveness of Transformer-based model for survival analysis in the industrial domain. Additionally, this work aims to explore the integration of Large Language Models as virtual assistants to facilitate user interaction and decision-making in the predictive maintenance process. The proposed system was tested on a public predictive maintenance problem and the Transformer-based model obtained better survival predictions than the baseline statistical and machine learning models, with improvements of up to 20%. Furthermore, the virtual assistant based on the large language model demonstrated the ability to retrieve precise information based on natural language prompts. |
|
![]() |
Topic: Annotation Guided Generative Adversarial Networks for Image Augmentation of Age-Related Macular Degeneration
Age-related macular degeneration (AMD) is a common eye disease that affects millions of people worldwide, leading to central vision loss. It mostly occurs in people over 50 years old. As per the literature, AMD diagnosis requires high-quality and diverse fundus images, which are often scarce and expensive to obtain. We propose a novel generative adversarial network (GAN) to address this challenge. It can synthesize realistic and clinically relevant images with AMD anomalies (drusen and Abnormalities of Choroidal Neovascular Membrane(CNVM)), enhancing dataset diversity and aiding automated diagnosis. Our GAN architecture is supplemented with (manual annotation guided) spatial attention maps to emphasize on anomalies guided through manually generated label maps. Further we propose 2 Siamese like Discriminators, for individual anomalies which emphasize the Generator module to generate AMD anomaly (drusen and abnormalities of CNVM) specific images. The proposed GAN leverages various loss functions, including contrastive loss and anomaly loss in the Siamese like Discriminator, and the adversarial loss in the combined model. The anomaly loss inspired by (Retinal detail Loss) measures the prominence of AMD signatures in the generated images. Our approach successfully generates synthetic images with specific AMD anomalies, aiding early diagnosis. The proposed GAN extended excellent dataset augmentation capabilities with 3% and 4% increase in diagnostic accuracy of drusen and abnormalities of CNVM, respectively. |
|
![]() |
Topic: Automation of Server Management Tasks and Data Aggregation using Generative AI
Machine learning has revolutionized industry practices by addressing complex operational challenges. Overheating server temperature due to administrative negligence at Institut Teknologi Sepuluh Nopember's Directorate of Technology and Information Systems Development (DPTSI) is a problem that can be solved with the help of machine learning. This research aims to create a user-friendly tool that enables DPTSI's server administrators to predict and prevent server-related problems effectively, thereby reducing downtime. The results of this study were obtained by the combination of iTransformer as a prediction model, Random Forest as a classification model, and Mistral-7B as a generative AI model deployed on a web-based application used to help server administrator decision-making abilities by predicting future server conditions and providing solutions on how to resolve predicted problems. This research contributes to the advancement of machine learning applications in industrial society, offering a practical solution for server management through innovative model combinations and implementation strategies. |
|
![]() |
Topic: Automated Crack Analysis and Reporting in Civil Infrastructure using Generative AI
Maintaining and inspecting infrastructure is crucial due to the safety hazards and economic costs of structural failures. Traditional methods are labor-intensive, time-consuming, and reactive. We propose an automated inspection system leveraging generative AI to enhance efficiency, predictive maintenance capabilities, and comprehensive data analysis. Our framework uses drone-based data acquisition with high-definition cameras and depth sensors, and a custom deep learning model, EyeNet, for precise crack detection. Generative AI techniques, including a Visual Question-Answering (VQA) model and an image-to-image model, are employed for detailed crack analysis and future crack pattern visualization, enabling proactive maintenance. The VQA model achieves an average Root Mean Square Error (RMSE) of 0.394 and an average Symmetric Mean Absolute Percentage Error (SMAPE) of 31.22%. A Large Language Model generates comprehensive reports with visualizations, accessible via a dedicated website. Our system significantly improves the inspection process compared to traditional methods, setting a new benchmark by combining generative AI for detailed crack analysis and predictive maintenance capabilities, creating a comprehensive inspection system for civil infrastructure. |
|
![]() |
Topic: AI-based Catalytic Performance Prediction for CO2 Electrochemical Reduction using Ionic Liquids
Recently, ionic liquids (ILs) have garnered remarkable attention as electrolytes for CO2 electrochemical reduction (CO2ER) due to their unique properties viz. thermal and chemical stability, good CO2 solubility, and their ability to reduce the reaction overpotential. The catalytic performance of ILs in CO2ER has been explored via experimental methods which have limitations, given the unclear understanding of the complex reaction mechanisms. Recently, Artificial Intelligence (AI) methods have gained increased attention across diverse applications including chemical engineering. These methods play a pivotal role in extracting insights, understanding patterns, and mitigating uncertainty within datasets. Hence, in this study, we investigate two categories of AI models (ML and DL) for predicting the CO2ER catalytic performance using ILs via Gibbs free energy and capacity. For this task, we formulate a novel dataset (CO2ERIL) from 90 ILs in two formats using: (1) 30 electronic and geometric IL properties, (2) IL chemical structures in SMILES data format. The dataset versions are formulated using the Conductor-like Screening Model for Realistic Solvents (COSMO-RS) and TURBOMOLE software. The prediction results of the AI models on the two dataset versions depict a similar trend in the predicted target variables signifying that the dataset format does not really affect the model performance. Moreover, the best ML model outperforms the DL model for one target variable, Free energy predictions, but lags in the other target variable, capacity predictions. Our study is novel in terms of AI methodology as well as the CO2ER dataset. Our research outcomes will contribute to the more effective selection of ILs for CO2ER catalysis. |
|
![]() |
Topic: Video Summarisation with Incident and Context Information using Generative AI
The proliferation of video content production has led to vast amounts of data, posing substantial challenges in terms of analysis efficiency and resource utilization. Addressing this issue calls for the development of robust video analysis tools. This paper proposes a novel approach leveraging Generative Artificial Intelligence (GenAI) to facilitate streamlined video analysis. Our tool aims to deliver tailored textual summaries of user-defined queries, offering a focused insight amidst extensive video datasets. Unlike conventional frameworks that offer generic summaries or limited action recognition, our method harnesses the power of GenAI to distil relevant information, enhancing analysis precision and efficiency. Employing YOLO-V8 for object detection and Gemini for comprehensive video and text analysis, our solution achieves heightened contextual accuracy. By combining YOLO with Gemini, our approach furnishes textual summaries extracted from extensive CCTV footage, enabling users to swiftly navigate and verify pertinent events without the need for exhaustive manual review. The quantitative evaluation revealed a similarity of 72.8%, while the qualitative assessment rated an accuracy of 85%, demonstrating the capability of the proposed method. |
|
![]() |
Topic: Generative AI for Improved Defect Detection in Semiconductor Wafers
Traditional methods for identifying defects in industrial settings are often complicated, inefficient, and rely heavily on expert knowledge for manual feature extraction and pipeline development. However, with the emergence of machine learning and deep learning, there's been a shift towards using these technologies for defect detection. Particularly, Generative AI has garnered attention for its ability to generate accurate defect identifications across various sectors. Our contribution lies in leveraging generative AI to revolutionize wafer manufacturing defect detection. We explore the balance between discriminative inference, known for its speed but susceptibility to shortcuts, and generative modeling, which offers enhanced robustness despite slower operation. Our investigation emphasizes the need for a model distinct from conventional discriminative ones, capable of adaptive learning to distinguish between normal and abnormal patterns in wafer manufacturing, aligning with expert judgment and utilizing historical data for predictive maintenance. We propose two innovative approaches using Generative AI: first, augmenting existing discriminative models with generative models for improved practicality; second, utilizing generative modeling to convert image-to-text models into classifiers. |