GEMINI API

Andreas suggested using Gemini Api with p5.js.

This code provides an interactive drawing experience that links visual input (the drawing) with auditory feedback based on sentiment, creating a multisensory link between visuals and sounds.

"Sentiment Analysis Based on Drawings, Mood-Based Soundscapes, and Interaction with Memory and Emotion."

✲ Drawing with a Mouse

• ꕀ To draw on the canvas by dragging the mouse, creating a freeform visual expression. The canvas is black initially, and the drawing appears in white lines.


✲ Sound Feedbacks based on Interpretation


• ꕀ When the user clicks the "Improv!" button or presses the space bar, the drawn visual is sent to the Gemini API. Here, the API provides a text-based "interpretation" or response to the visual.


• ꕀ Keywords in the API’s response (e.g., "happy," "sad," "calm"), triggers a corresponding ambient hum sound with a frequency tailored to the mood.


✲ Interactive Experience


• ꕀ Engaging Interface - (create a theme for a more engaging experience where participants can draw their memories or emotions as shapes while software responds by playing relevant sounds)


✲ Real-Time Text Response and Display


• ꕀ The Gemini API response is displayed on the canvas, adding a layer of interaction, showing the sentiment or mood description with a background sound, making it an audio-visual experience.

TESTING AND TWEAKING THE CODE

slightly changing the prompts generates different text readings

doing this multiple times until I get my desired outcome

also added a code to generate a random frequency - low, medium or high, to enhance the whole audio-visual experience